Why does reading a file with TextFile.input() give me unset lines?

← Security and Sanitization · Ref: Q1370

TextFile.input() screens every line as it is read. A line that trips a threat detector is REJECTED and comes back as an unset String - it is not passed on. This is the secure default. Use unsanitizedInput() when the file legitimately contains content that looks dangerous.

SCREENED READ - THE DEFAULT

file.input() wraps the read so each line passes through InputSanitizer:

  try
    -> input <- file.input()
    while input.hasNext()
      line <- input.next()
      if line?
        stdout.println(line)
      else
        stdout.println("blocked")

Always test the line with line? before using it. A rejected line is unset.

UNSET MEANS REJECTED, NOTHING ELSE

While hasNext() is true, an unset line can only mean the line was rejected. A blank line or a whitespace-only line reads back SET (an empty or blank String is a set String), so there is never any confusion between "rejected" and "empty".

UNSCREENED READ - THE BYPASS

Some files legitimately contain what looks like an attack: SQL migration scripts, HTML fragments, shell scripts, log files containing --. Screening those would discard real data, so read them with unsanitizedInput():

  try
    -> rawInput <- file.unsanitizedInput()
    while rawInput.hasNext()
      line <- rawInput.next()
      stdout.println(line)

Every line passes through unchanged.

GETTING THE REASON, NOT JUST THE FACT

input() tells you a line was rejected but not why - the threat category goes to the sanitizer log. When the program itself needs the category, read raw and ask InputSanitizer directly:

  sanitizer <- InputSanitizer()
  threat <- sanitizer.detectThreat(line)
  if threat?
    stdout.println(threat)

detectThreat() returns a comma separated list such as "SQL_INJECTION, SQL_INJECTION_SIGNATURE" and, unlike sanitize(), does not write to the log. It returns an unset String when the line is safe.

SCREENING IS A JUDGEMENT, NOT A PROOF

Detection is heuristic in both directions. That is why the bypass exists, and why the real guarantee comes from the compiler-enforced sanitized modifier at the point a String enters a dangerous context - not from the file read.

See Q151 for reading text files. See Q1237 for sanitized function parameters. See Q1246 for sanitized constructor parameters. See Q1247 for sanitized method parameters.

Example

defines module qa.security.sanitizedread

  defines program

    SanitizedVsUnsanitizedRead()
      stdout <- Stdout()

      file <- TextFile("/tmp/qa-example-mixed.txt")
      stdout.println("File: " + $file)

      // === SCREENED READ - THE DEFAULT ===

      // input() screens every line. A rejected line comes back UNSET,
      // so always test it with line? before using the value.
      stdout.println("Screened read")
      try
        -> input <- file.input()
        while input.hasNext()
          line <- input.next()
          if line?
            stdout.println(line)
          else
            stdout.println("blocked")

      // === UNSCREENED READ - THE BYPASS ===

      // Use unsanitizedInput() for files that legitimately hold content
      // resembling an attack: SQL scripts, HTML, shell scripts, logs with --
      stdout.println("Unscreened read")
      try
        -> rawInput <- file.unsanitizedInput()
        while rawInput.hasNext()
          line <- rawInput.next()
          stdout.println(line)

      // === THE REASON, NOT JUST THE FACT ===

      // input() reports THAT a line was rejected; detectThreat reports WHY.
      // It returns an unset String for safe content and does not write a log entry.
      sanitizer <- InputSanitizer()
      stdout.println("Threat categories")
      try
        -> rawInput <- file.unsanitizedInput()
        while rawInput.hasNext()
          line <- rawInput.next()
          threat <- sanitizer.detectThreat(line)
          if threat?
            stdout.println(threat)

Common mistakes

E50060 — input() screens each line, so a rejected line is unset. Test it with line? before use, or read via unsanitizedInput() if the content is trusted. See ek9 -h E50060 for details.

Incorrect:

line <- input.next()
      stdout.println(line)

Correct:

line <- input.next()
      if line?
        stdout.println(line)

Coming from another language?

Java/Python/Go: reading a file returns bytes or lines with no screening at all; any input validation is something you remember to add. EK9: TextFile.input() screens by default and reports a rejection as an unset String, so forgetting to check is visible rather than silent. unsanitizedInput() is the deliberate, named bypass, and InputSanitizer.detectThreat() supplies the threat category when the program needs to branch on it.

Keywords: security, injection, textfile, file, bypass, rejected, sanitize, unsanitized, unset, threat, read, blocked, input