epan: Register dynamic column fields and make them filterable

Make the text of each registered column a FT_STRING field that can be
filtered, prefixed with _ws.col - these work in display filters, filters
in taps, coloring rules, Wireshark read filters, and in the -Y, -R, -e,
and -j options to tshark. Use them as the default "Apply as Filter" value
for the columns that aren't handled by anything else currently.

Because only the columns formats that actually correspond to columns
get filled in (invisible columns work), register and deregister the
fields when the columns change.

Use the lower case version of the rest of the COL_* define for each
column as the field name.

This adds a number of conditions to "when are the columns needed",
including when the main display filter or any filter on a tap is
using one of these fields.

Custom columns are currently not implemented. For custom columns, the
tree then has to be further primed with any fields used by the custom
columns as well. (Perhaps that should happen in epan_dissect_run() -
are there any cases where we construct the columns and don't want to
prime with any field that custom columns contains? Possibly in taps
that we know only use build in columns.)

Thus, for performance reasons, you're better off matching an ordinary
field if possible; it takes extra time to generate the columns and many
of them are numeric types. (Note that you can always convert a non-string
field to a string field if you want regex matching, consult the
*wireshark-filter(4)* man page.) It does save a bit on typing (especially
for a multifield custom column) and remembering the column title might
be easier in some cases.

The columns are set before the color filters, which means that you
can have a color filter that depends on a built-in column like Info or
Protocol.

Remove the special handling for the -e option to tshark. Note that
the behavior is a little different now, because fixed field names
are used instead of the titles (using the titles allowed illegal
filter names, because it wasn't going through the filter engine.)
For default names, this means that they're no longer capitalized,
so "_ws.col.info" instead of "_ws.col.Info" - hopefully a small
price in exchange for the filters working everywhere.

The output format for -T fields remains the same; all that special
handling is removed (except for remembering if someone asked for
a column field to know that columns should be constructed.)

They're also set before the postdissectors, so postdissectors can
have access.

Anything that depends on whether a packet and previous packets are
displayed (COL_DELTA_TIME_DIS or COL_CUMULATIVE_BYTES) doesn't work
the way most people expect, so don't register fields for those.
(The same is already true of color filters that use those, along with
color filters that use the color filter fields.)

Fix #16576. Fix #17971. Fix #4684. Fix #13491. Fix #13941.

2 files changed, 80 insertions, 5 deletions

diff --git a/test/suite_dfilter/dfiltertest.py b/test/suite_dfilter/dfiltertest.py
index 6a92c8f337..a28df64ce2 100644
--- a/test/suite_dfilter/dfiltertest.py
+++ b/test/suite_dfilter/dfiltertest.py
@@ -8,7 +8,7 @@ import pytest
 
 @pytest.fixture
 def dfilter_cmd(cmd_tshark, capture_file, request):
-    def wrapped(dfilter, frame_number=None, prefs=None):
+    def wrapped(dfilter, frame_number=None, prefs=None, read_filter=False):
         cmd = [
             cmd_tshark,
             "-n",       # No name resolution
@@ -20,10 +20,17 @@ def dfilter_cmd(cmd_tshark, capture_file, request):
                 "-2",       # two-pass mode
                 "--selected-frame={}".format(frame_number)
             ])
-        cmd.extend([
-            "-Y",       # packet display filter (used to be -R)
-            dfilter
-        ])
+        if read_filter:
+            cmd.extend([
+                "-2",       # two-pass mode
+                "-R",       # read filter (requires two-pass mode)
+                dfilter
+            ])
+        else:
+            cmd.extend([
+                "-Y",       # packet display filter (used to be -R)
+                dfilter
+            ])
         if prefs:
             cmd.extend([
             "-o",
@@ -67,6 +74,20 @@ def checkDFilterCountWithSelectedFrame(dfilter_cmd, base_env):
         assert dfp_count == expected_count, msg
     return checkDFilterCount_real
 
+@pytest.fixture
+def checkDFilterCountReadFilter(dfilter_cmd, base_env):
+    def checkDFilterCount_real(dfilter, expected_count):
+        """Run a read filter in two pass mode and expect a certain number of packets."""
+        output = subprocess.check_output(dfilter_cmd(dfilter, read_filter=True),
+                                         universal_newlines=True,
+                                         stderr=subprocess.STDOUT,
+                                         env=base_env)
+
+        dfp_count = output.count("\n")
+        msg = "Expected %d, got: %s\noutput: %r" % \
+            (expected_count, dfp_count, output)
+        assert dfp_count == expected_count, msg
+    return checkDFilterCount_real
 
 @pytest.fixture
 def checkDFilterFail(cmd_dftest, base_env):
diff --git a/test/suite_dfilter/group_columns.py b/test/suite_dfilter/group_columns.py
new file mode 100644
index 0000000000..5bbfbc90c9
--- /dev/null
+++ b/test/suite_dfilter/group_columns.py
@@ -0,0 +1,54 @@
+# Copyright (c) 2013 by Gilbert Ramirez <gram@alumni.rice.edu>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import pytest
+from suite_dfilter.dfiltertest import *
+
+class TestDfilterColumns:
+    trace_file = "http.pcap"
+
+    def test_exists_1(self, checkDFilterCount):
+        dfilter = "_ws.col.info"
+        checkDFilterCount(dfilter, 1)
+
+    def test_exists_2(self, checkDFilterFail):
+        # Column not in the default configuration
+        dfilter = "_ws.col.expert"
+        error = f'"{dfilter}" is not a valid protocol or protocol field'
+        checkDFilterFail(dfilter, error)
+
+    def test_exists_3(self, checkDFilterFail):
+        # Column not registered as field (it behaves unusally if filtered)
+        dfilter = "_ws.col.delta_time_dis"
+        error = f'"{dfilter}" is not a valid protocol or protocol field'
+        checkDFilterFail(dfilter, error)
+
+    def test_func_1(self, checkDFilterCount):
+        dfilter = "len(_ws.col.protocol) == 4"
+        checkDFilterCount(dfilter, 1)
+
+    def test_matches_1(self, checkDFilterSucceed):
+        dfilter = '_ws.col.info matches "^HEAD"'
+        checkDFilterSucceed(dfilter)
+
+    def test_equal_1(self, checkDFilterCount):
+        dfilter = '_ws.col.protocol == "HTTP"'
+        checkDFilterCount(dfilter, 1)
+
+    def test_equal_2(self, checkDFilterCount):
+        dfilter = '_ws.col.def_dst == "207.46.134.94"'
+        checkDFilterCount(dfilter, 1)
+
+    def test_not_equal_1(self, checkDFilterCount):
+        dfilter = '_ws.col.def_src != "10.0.0.5"'
+        checkDFilterCount(dfilter, 0)
+
+    def test_read_filter(self, checkDFilterCountReadFilter):
+        dfilter = '_ws.col.protocol == "HTTP"'
+        checkDFilterCountReadFilter(dfilter, 1)
+
+    def test_add_column(self, checkDFilterCount):
+        # Add column to configuration
+        dfilter = '_ws.col.expert == "Chat"'
+        checkDFilterCount(dfilter, 1, 'gui.column.format:"Expert","%a"')