libctf, include: find types of symbols by name

The existing ctf_lookup_by_symbol and ctf_arc_lookup_symbol functions
suffice to look up the types of symbols if the caller already has a
symbol number.  But the caller often doesn't have one of those and only
knows the name of the symbol: also, in object files, the caller might
not have a useful symbol number in any sense (and neither does libctf:
the 'symbol number' we use in that case literally starts at 0 for the
lexicographically first-sorted symbol in the symtypetab and counts those
symbols, so it corresponds to nothing useful).

This means that even though object files have a symtypetab (generated by
the compiler or by ld -r), the only way we can look up anything in it is
to iterate over all symbols in turn with ctf_symbol_next until we find
the one we want.

This is unhelpful and pointlessly inefficient.

So add a pair of functions to look up symbols by name in a dict and in a
whole archive: ctf_lookup_by_symbol_name and ctf_arc_lookup_symbol_name.
These are identical to the existing functions except that they take
symbol names rather than symbol numbers.

To avoid insane repetition, we do some refactoring in the process, so
that both ctf_lookup_by_symbol and ctf_arc_lookup_symbol turn into thin
wrappers around internal functions that do both lookup by symbol index
and lookup by name.  This massively reduces code duplication because
even the existing lookup-by-index stuff wants to use a name sometimes
(when looking up in indexed sections), and the new lookup-by-name stuff
has to turn it into an index sometimes (when looking up in non-indexed
sections): doing it this way lets us share most of that.

The actual name->index lookup is done by ctf_lookup_symbol_idx.  We do
not anticipate this lookup to be as heavily used as ld.so symbol lookup
by many orders of magnitude, so using the ELF symbol hashes would
probably take more time to read them than is saved by using the hashes,
and it adds a lot of complexity.  Instead, do a linear search for the
symbol name, caching all the name -> index mappings as we go, so that
future searches are likely to hit in the cache.  To avoid having to
repeat this search over and over in a CTF archive when
ctf_arc_lookup_symbol_name is used, have cached archive lookups (the
sort done by ctf_arc_lookup_symbol* and the ctf_archive_next iterator)
pick out the first dict they cache in a given archive and store it in a
new ctf_archive field, ctfi_crossdict_cache.  This can be used to store
cross-dictionary cached state that depends on things like the ELF symbol
table rather than the contents of any one dict.  ctf_lookup_symbol_idx
then caches its name->index mappings in the dictionary named in the
crossdict cache, if any, so that ctf_lookup_symbol_idx in other dicts
in the same archive benefit from the previous linear search, and the
symtab only needs to be scanned at most once.

(Note that if you call ctf_lookup_by_symbol_name in one specific dict,
and then follow it with a ctf_arc_lookup_symbol_name, the former will
not use the crossdict cache because it's only populated by the dict
opens in ctf_arc_lookup_symbol_name. This is harmless except for a small
one-off waste of memory and time: it's only a cache, after all.  We can
fix this later by using the archive caching machinery more
aggressively.)

In ctf-archive, we do similar things, turning ctf_arc_lookup_symbol into
a wrapper around a new function that does both index -> ID and name ->
ID lookups across all dicts in an archive.  We add a new
ctfi_symnamedicts cache that maps symbol names to the ctf_dict_t * that
it was found in (so that linear searches for symbols don't need to be
repeated): but we also *remove* a cache, the ctfi_syms cache that was
memoizing the actual ctf_id_t returned from every call to
ctf_arc_lookup_symbol.  This is pointless: all it saves is one call to
ctf_lookup_by_symbol, and that's basically an array lookup and nothing
more so isn't worth caching.  (Equally, given that symbol -> index
mappings are cached by ctf_lookup_by_symbol_name, those calls are nearly
free after the first call, so there's no point caching the ctf_id_t in
that case either.)

We fix up one test that was doing manual symbol lookup to use
ctf_arc_lookup_symbol instead, and enhance it to check that the caching
layer is not totally broken: we also add a new test to do lookups in a
.o file, and another to do lookups in an archive with conflicted types
and make sure that sort of multi-dict lookup is actually working.

include/ChangeLog
2021-02-17  Nick Alcock  <nick.alcock@oracle.com>

	* ctf-api.h (ctf_arc_lookup_symbol_name): New.
	(ctf_lookup_by_symbol_name): Likewise.

libctf/ChangeLog
2021-02-17  Nick Alcock  <nick.alcock@oracle.com>

	* ctf-impl.h (ctf_dict_t) <ctf_symhash>: New.
	<ctf_symhash_latest>: Likewise.
	(struct ctf_archive_internal) <ctfi_crossdict_cache>: New.
	<ctfi_symnamedicts>: New.
	<ctfi_syms>: Remove.
	(ctf_lookup_symbol_name): Remove.
	* ctf-lookup.c (ctf_lookup_symbol_name): Propagate errors from
	parent properly.  Make static.
	(ctf_lookup_symbol_idx): New, linear search for the symbol name,
	cached in the crossdict cache's ctf_symhash (if available), or
	this dict's (otherwise).
	(ctf_try_lookup_indexed): Allow the symname to be passed in.
	(ctf_lookup_by_symbol): Turn into a wrapper around...
	(ctf_lookup_by_sym_or_name): ... this, supporting name lookup too,
	using ctf_lookup_symbol_idx in non-writable dicts.  Special-case
	name lookup in dynamic dicts without reported symbols, which have
	no symtab or dynsymidx but where name lookup should still work.
	(ctf_lookup_by_symbol_name): New, another wrapper.
	* ctf-archive.c (enosym): Note that this is present in
	ctfi_symnamedicts too.
	(ctf_arc_close): Adjust for removal of ctfi_syms.  Free the
	ctfi_symnamedicts.
	(ctf_arc_flush_caches): Likewise.
	(ctf_dict_open_cached): Memoize the first cached dict in the
	crossdict cache.
	(ctf_arc_lookup_symbol): Turn into a wrapper around...
	(ctf_arc_lookup_sym_or_name): ... this.  No longer cache
	ctf_id_t lookups: just call ctf_lookup_by_symbol as needed (but
	still cache the dicts those lookups succeed in).  Add
	lookup-by-name support, with dicts of successful lookups cached in
	ctfi_symnamedicts.  Refactor the caching code a bit.
	(ctf_arc_lookup_symbol_name): New, another wrapper.
	* ctf-open.c (ctf_dict_close): Free the ctf_symhash.
	* libctf.ver (LIBCTF_1.2): New version.  Add
	ctf_lookup_by_symbol_name, ctf_arc_lookup_symbol_name.
	* testsuite/libctf-lookup/enum-symbol.c (main): Use
	ctf_arc_lookup_symbol rather than looking up the name ourselves.
	Fish it out repeatedly, to make sure that symbol caching isn't
	broken.
	(symidx_64): Remove.
	(symidx_32): Remove.
	* testsuite/libctf-lookup/enum-symbol-obj.lk: Test symbol lookup
	in an unlinked object file (indexed symtypetab sections only).
	* testsuite/libctf-writable/symtypetab-nonlinker-writeout.c
	(try_maybe_reporting): Check symbol types via
	ctf_lookup_by_symbol_name as well as ctf_symbol_next.
	* testsuite/libctf-lookup/conflicting-type-syms.*: New test of
	lookups in a multi-dict archive.

7 files changed, 169 insertions, 101 deletions

diff --git a/libctf/testsuite/libctf-lookup/conflicting-type-syms-a.c b/libctf/testsuite/libctf-lookup/conflicting-type-syms-a.c
new file mode 100644
index 00000000000..65414877114
--- /dev/null
+++ b/libctf/testsuite/libctf-lookup/conflicting-type-syms-a.c
@@ -0,0 +1,5 @@
+typedef long a_t;
+typedef long b_t;
+
+a_t *a;
+b_t ignore2;
diff --git a/libctf/testsuite/libctf-lookup/conflicting-type-syms-b.c b/libctf/testsuite/libctf-lookup/conflicting-type-syms-b.c
new file mode 100644
index 00000000000..e458021efb1
--- /dev/null
+++ b/libctf/testsuite/libctf-lookup/conflicting-type-syms-b.c
@@ -0,0 +1,5 @@
+typedef long a_t;
+typedef long b_t;
+
+a_t b;
+b_t ignore1;
diff --git a/libctf/testsuite/libctf-lookup/conflicting-type-syms.c b/libctf/testsuite/libctf-lookup/conflicting-type-syms.c
new file mode 100644
index 00000000000..ffe6983e250
--- /dev/null
+++ b/libctf/testsuite/libctf-lookup/conflicting-type-syms.c
@@ -0,0 +1,99 @@
+#include "config.h"
+#include <ctf-api.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+int
+main (int argc, char *argv[])
+{
+  ctf_archive_t *ctf;
+  ctf_dict_t *a_fp, *ignore1_fp, *b_fp, *ignore2_fp, *tmp_fp;
+  int err;
+  ctf_id_t a, b, ignore1, ignore2, tmp;
+  char *foo;
+  ctf_next_t *i = NULL;
+  const char *name;
+  int val;
+
+  if (argc != 2)
+    {
+      fprintf (stderr, "Syntax: %s PROGRAM\n", argv[0]);
+      exit(1);
+    }
+
+  if ((ctf = ctf_open (argv[1], NULL, &err)) == NULL)
+    goto open_err;
+
+  /* Fish out each symbol in turn: also try to fish out a nonexistent one.  */
+
+  if ((a_fp = ctf_arc_lookup_symbol_name (ctf, "a", &a, &err)) == NULL)
+    goto sym_err;
+  printf ("Type of a is %s\n", foo = ctf_type_aname (a_fp, a));
+
+  if ((b_fp = ctf_arc_lookup_symbol_name (ctf, "b", &b, &err)) == NULL)
+    goto sym_err;
+  printf ("Type of b is %s\n", foo = ctf_type_aname (b_fp, b));
+
+  if ((ignore1_fp = ctf_arc_lookup_symbol_name (ctf, "ignore1", &ignore1, &err)) == NULL)
+    goto sym_err;
+  printf ("Type of ignore1 is %s\n", foo = ctf_type_aname (ignore1_fp, ignore1));
+
+  if ((ignore2_fp = ctf_arc_lookup_symbol_name (ctf, "ignore2", &ignore2, &err)) == NULL)
+    goto sym_err;
+  printf ("Type of ignore2 is %s\n", foo = ctf_type_aname (ignore2_fp, ignore1));
+
+  /* Try a call in just-get-the-dict mode and make sure it doesn't fail.  */
+  if ((tmp_fp = ctf_arc_lookup_symbol_name (ctf, "ignore2", NULL, &err)) == NULL)
+    goto sym_err;
+  ctf_dict_close (tmp_fp);
+
+  /* Make sure failures fail.  */
+  if  ((ctf_arc_lookup_symbol_name (ctf, "nonexistent", NULL, &err) != NULL)
+       || err != ECTF_NOTYPEDAT)
+    goto nosym_err;
+
+  /* Fish them out again to check the caching layer.  */
+  if (((tmp_fp = ctf_arc_lookup_symbol_name (ctf, "a", &tmp, &err)) != a_fp)
+      || (tmp != a))
+    goto sym_cache_err;
+  ctf_dict_close (tmp_fp);
+
+  if (((tmp_fp = ctf_arc_lookup_symbol_name (ctf, "b", &tmp, &err)) != b_fp)
+      || (tmp != b))
+    goto sym_cache_err;
+  ctf_dict_close (tmp_fp);
+
+  if (((tmp_fp = ctf_arc_lookup_symbol_name (ctf, "ignore1", &tmp, &err)) != ignore1_fp)
+      || (tmp != ignore1))
+    goto sym_cache_err;
+  ctf_dict_close (tmp_fp);
+
+  if (((tmp_fp = ctf_arc_lookup_symbol_name (ctf, "ignore2", &tmp, &err)) != ignore2_fp)
+      || (tmp != ignore2))
+    goto sym_cache_err;
+  ctf_dict_close (tmp_fp);
+
+  ctf_dict_close (a_fp);
+  ctf_dict_close (b_fp);
+  ctf_dict_close (ignore1_fp);
+  ctf_dict_close (ignore2_fp);
+  ctf_close (ctf);
+
+  return 0;
+
+ open_err:
+  fprintf (stderr, "%s: cannot open: %s\n", argv[0], ctf_errmsg (err));
+  return 1;
+ sym_err:
+  fprintf (stderr, "%s: Symbol lookup error: %s\n", argv[0], ctf_errmsg (err));
+  return 1;
+ nosym_err:
+  fprintf (stderr, "%s: Nonexistent symbol lookup unexpected error: %s\n", argv[0],
+	   ctf_errmsg (err));
+  return 1;
+ sym_cache_err:
+  fprintf (stderr, "%s: Symbol re-lookup error (caching bug): %s\n", argv[0],
+	   ctf_errmsg (err));
+  return 1;
+}
diff --git a/libctf/testsuite/libctf-lookup/conflicting-type-syms.lk b/libctf/testsuite/libctf-lookup/conflicting-type-syms.lk
new file mode 100644
index 00000000000..09b41dfd623
--- /dev/null
+++ b/libctf/testsuite/libctf-lookup/conflicting-type-syms.lk
@@ -0,0 +1,7 @@
+# lookup: conflicting-type-syms.c
+# source: conflicting-type-syms-a.c
+# source: conflicting-type-syms-b.c
+Type of a is a_t \*
+Type of b is a_t
+Type of ignore1 is b_t
+Type of ignore2 is b_t
diff --git a/libctf/testsuite/libctf-lookup/enum-symbol-obj.lk b/libctf/testsuite/libctf-lookup/enum-symbol-obj.lk
new file mode 100644
index 00000000000..548d67b70c6
--- /dev/null
+++ b/libctf/testsuite/libctf-lookup/enum-symbol-obj.lk
@@ -0,0 +1,5 @@
+# lookup: enum-symbol.c
+# source: enum-symbol-ctf.c
+red1 has value 0
+green1 has value 1
+blue1 has value 2
diff --git a/libctf/testsuite/libctf-lookup/enum-symbol.c b/libctf/testsuite/libctf-lookup/enum-symbol.c
index 4f63b61b803..c67478fe780 100644
--- a/libctf/testsuite/libctf-lookup/enum-symbol.c
+++ b/libctf/testsuite/libctf-lookup/enum-symbol.c
@@ -1,71 +1,19 @@
 #include "config.h"
-#include <bfd.h>
-#include <elf.h>
 #include <ctf-api.h>
-#include <swap.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 
-ssize_t symidx_64 (ctf_sect_t *s, ctf_sect_t *strsect, int little_endian, const char *name)
-{
-  const char *strs = (const char *) strsect->cts_data;
-  size_t i;
-  Elf64_Sym *sym = (Elf64_Sym *) s->cts_data;
-  for (i = 0; i < s->cts_size / s->cts_entsize; i++, sym++)
-    {
-      Elf64_Word nameoff = sym->st_name;
-#ifdef WORDS_BIGENDIAN
-      if (little_endian)
-	swap_thing (nameoff);
-#else
-      if (!little_endian)
-	swap_thing (nameoff);
-#endif
-      if (strcmp (strs + nameoff, name) == 0)
-	return i;
-    }
-  return -1;
-}
-
-ssize_t symidx_32 (ctf_sect_t *s, ctf_sect_t *strsect, int little_endian, const char *name)
-{
-  const char *strs = (const char *) strsect->cts_data;
-  size_t i;
-  Elf32_Sym *sym = (Elf32_Sym *) s->cts_data;
-  for (i = 0; i < s->cts_size / s->cts_entsize; i++, sym++)
-    {
-      Elf32_Word nameoff = sym->st_name;
-#ifdef WORDS_BIGENDIAN
-      if (little_endian)
-	swap_thing (nameoff);
-#else
-      if (!little_endian)
-	swap_thing (nameoff);
-#endif
-      if (strcmp (strs + nameoff, name) == 0)
-	return i;
-    }
-  return -1;
-}
-
 int
 main (int argc, char *argv[])
 {
-  ctf_dict_t *fp;
-  bfd *abfd;
   ctf_archive_t *ctf;
-  ctf_sect_t symsect;
-  ctf_sect_t strsect;
-  ssize_t symidx;
+  ctf_dict_t *fp, *tmp_fp;
   int err;
-  ctf_id_t type;
+  ctf_id_t type, tmp;
   ctf_next_t *i = NULL;
   const char *name;
   int val;
-  int little_endian;
-
-  ssize_t (*get_sym) (ctf_sect_t *s, ctf_sect_t *strsect, int little_endian, const char *name);
 
   if (argc != 2)
     {
@@ -73,53 +21,12 @@ main (int argc, char *argv[])
       exit(1);
     }
 
-  /* Figure out the endianness of the symtab(s).  */
-  if ((abfd = bfd_openr (argv[1], NULL)) == NULL
-      || !bfd_check_format (abfd, bfd_object))
-    goto bfd_open_err;
-  little_endian = bfd_little_endian (abfd);
-  bfd_close_all_done (abfd);
-
   if ((ctf = ctf_open (argv[1], NULL, &err)) == NULL)
     goto open_err;
 
-  if ((fp = ctf_dict_open (ctf, NULL, &err)) == NULL)
-    goto open_err;
-
-  symsect = ctf_getsymsect (fp);
-  strsect = ctf_getstrsect (fp);
-  if (symsect.cts_data == NULL
-      || strsect.cts_data == NULL)
-    {
-      fprintf (stderr, "%s: no symtab or strtab\n", argv[0]);
-      return 1;
-    }
-
-  ctf_dict_close (fp);
-
-  if (symsect.cts_entsize != sizeof (Elf64_Sym) &&
-      symsect.cts_entsize != sizeof (Elf32_Sym))
-    {
-      fprintf (stderr, "%s: unknown symsize: %lx\n", argv[0],
-	       symsect.cts_size);
-      return 1;
-    }
-
-  switch (symsect.cts_entsize)
-    {
-    case sizeof (Elf64_Sym): get_sym = symidx_64; break;
-    case sizeof (Elf32_Sym): get_sym = symidx_32; break;
-    }
-
-  if ((symidx = get_sym (&symsect, &strsect, little_endian, "primary1")) < 0)
-    {
-      fprintf (stderr, "%s: symbol not found: primary1\n", argv[0]);
-      return 1;
-    }
-
-  /* Fish it out, then fish out all its enumerand/value pairs.  */
+  /* Fish out the enumerator, then fish out all its enumerand/value pairs.  */
 
-  if ((fp = ctf_arc_lookup_symbol (ctf, symidx, &type, &err)) == NULL)
+  if ((fp = ctf_arc_lookup_symbol_name (ctf, "primary1", &type, &err)) == NULL)
     goto sym_err;
 
   while ((name = ctf_enum_next (fp, type, &i, &val)) != NULL)
@@ -129,21 +36,27 @@ main (int argc, char *argv[])
   if (ctf_errno (fp) != ECTF_NEXT_END)
     goto nerr;
 
+  /* Fish it out again to check the caching layer.  */
+  if (((tmp_fp = ctf_arc_lookup_symbol_name (ctf, "primary1", &tmp, &err)) != fp)
+      || (tmp != type))
+    goto sym_cache_err;
+
+  ctf_dict_close (tmp_fp);
   ctf_dict_close (fp);
   ctf_close (ctf);
 
   return 0;
 
- bfd_open_err:
-  fprintf (stderr, "%s: cannot open: %s\n", argv[0], bfd_errmsg (bfd_get_error ()));
-  return 1;
-
  open_err:
   fprintf (stderr, "%s: cannot open: %s\n", argv[0], ctf_errmsg (err));
   return 1;
  sym_err:
   fprintf (stderr, "%s: Symbol lookup error: %s\n", argv[0], ctf_errmsg (err));
   return 1;
+ sym_cache_err:
+  fprintf (stderr, "%s: Symbol re-lookup error (caching bug): %s\n", argv[0],
+	   ctf_errmsg (err));
+  return 1;
  err:
   fprintf (stderr, "Lookup failed: %s\n", ctf_errmsg (ctf_errno (fp)));
   return 1;
diff --git a/libctf/testsuite/libctf-writable/symtypetab-nonlinker-writeout.c b/libctf/testsuite/libctf-writable/symtypetab-nonlinker-writeout.c
index d339963d264..98144de6391 100644
--- a/libctf/testsuite/libctf-writable/symtypetab-nonlinker-writeout.c
+++ b/libctf/testsuite/libctf-writable/symtypetab-nonlinker-writeout.c
@@ -98,6 +98,21 @@ try_maybe_reporting (int report)
   if (ctf_errno (fp) != ECTF_NEXT_END)
     goto iter_err;
 
+  /* Look up all the symbols by name and make sure that works.  */
+
+  if (ctf_lookup_by_symbol_name (fp, "data_a") != base2)
+    goto lookup_syms_err;
+  if (ctf_lookup_by_symbol_name (fp, "data_b") != base3)
+    goto lookup_syms_err;
+  if (ctf_lookup_by_symbol_name (fp, "data_c") != base)
+    goto lookup_syms_err;
+  if (ctf_lookup_by_symbol_name (fp, "func_a") != func2)
+    goto lookup_syms_err;
+  if (ctf_lookup_by_symbol_name (fp, "func_b") != func3)
+    goto lookup_syms_err;
+  if (ctf_lookup_by_symbol_name (fp, "func_c") != func)
+    goto lookup_syms_err;
+
   /* Possibly report some but not all of the symbols, as if we are a linker (no
      real program would do this without using the ctf_link APIs, but it's not
      *prohibited*, just useless, and if they do we don't want things to
@@ -121,6 +136,21 @@ try_maybe_reporting (int report)
       if (report_sym (fp, &sym, "func_c", 4, 2) < 0 ||
 	  report_sym (fp, &sym, "func_a", 5, 2) < 0)
 	goto report_err;
+
+      /* Look up all the symbols by name now we have reported symbols.  */
+
+      if (ctf_lookup_by_symbol_name (fp, "data_a") != base2)
+	goto lookup_syms_err;
+      if (ctf_lookup_by_symbol_name (fp, "data_b") != base3)
+	goto lookup_syms_err;
+      if (ctf_lookup_by_symbol_name (fp, "data_c") != base)
+	goto lookup_syms_err;
+      if (ctf_lookup_by_symbol_name (fp, "func_a") != func2)
+	goto lookup_syms_err;
+      if (ctf_lookup_by_symbol_name (fp, "func_b") != func3)
+	goto lookup_syms_err;
+      if (ctf_lookup_by_symbol_name (fp, "func_c") != func)
+	goto lookup_syms_err;
     }
 
   /* Write out, to memory.  */
@@ -203,6 +233,10 @@ try_maybe_reporting (int report)
  expected_overshoot_err:
   fprintf (stderr, "Too many symbols in post-writeout comparison\n");
   exit (1);
+ lookup_syms_err:
+  fprintf (stderr, "Explicit lookup of symbols by name failed: %s\n",
+	   ctf_errmsg (ctf_errno (fp)));
+  exit (1);
  expected_compar_err:
   fprintf (stderr, "Non-dynamic iteration comparison failure: %s "
 	   "(type %lx): expected %s (type %lx)\n", symname, symtype,