| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This will be useful to handle latitude and longitude numbers for GAD, which is
the location estimate representation used for LCS (Location Services).
The OsmoSMLC VTY user interface will provide floating-point strings like
"23.456" while GAD stores them as micro-degress 23456000. The osmo_gad_to_str*
will also convert latitude and longitude to floating-point string.
There was code review concerns against adding this API, upon which I tried to
use floating point string formats. But I encountered various problems with
accuracy and trailing zeros. For global positioning data (latitude and
longitude), even inaccuracy on the sixth significant decimal digit causes
noticeable positional shift. To achieve sufficient accuracy on the least
significant end, I need to use double instead of float. To remove trailing
zeros, the idea was to use '%.6g' format, but that can cause rounding. '%.6f'
on a double looks ok, but always includes trailing zeros. A test program shows:
%.6g of ((double)(int32_t)23230100)/1e6 = "23.2301" <-- good
%.6g of ((double)(int32_t)42419993)/1e6 = "42.42" <-- bad rounding
%.6g of ((double)(int32_t)23230199)/1e6 = "23.2302" <-- bad rounding
%.6f of ((double)(int32_t)23230100)/1e6 = "23.230100" <-- trailing zeros
%.6f of ((double)(int32_t)42419993)/1e6 = "42.419993" <-- good
%.6f of ((double)(int32_t)23230199)/1e6 = "23.230199" <-- good
It looks like when accepting that there will be trailing zeros, using double
with '%.6f' would work out, but in the end I am not certain enough that there
aren't more hidden rounding / precision glitches. Hence I decided to reinforce
the need to add this API: it is glitch free in sufficient precision for
latitude and longitude data, because it is based on integer arithmetic.
The need for this precision is particular to the (new) OsmoSMLC vty
configuration, where reading and writing back user config must not modify the
values the user entered. Considering to add these functions to osmo-smlc.git,
we might as well add them here to libosmocore utils, and also use them in
osmo_gad_to_str_*() functions.
Change-Id: Ib9aee749cd331712a4dcdadfb6a2dfa4c26da957
|
|
Related: SYS#4915
Change-Id: I439c7fa52a3a30eebc3d35e78be7f1724fb69294
|
|
Implement better API around 3GPP TS 24.008 Mobile Identity coding.
struct osmo_mobile_identity is a decoded representation of the raw Mobile
Identity, with a string representation as well as dedicated raw uint32_t TMSI.
The aim is to remove all uncertainty about decoded buffer sizes / data types.
I have patches ready for current osmo CNI programs, replacing the Mobile
Identity coding with this new API. Deprecate the old MI API.
osmo-bsc: I71c3b4c65dbfdfa51409e09d4868aea83225338a
osmo-msc: Ic3f969e739654c1e8c387aedeeba5cce07fe2307
osmo-sgsn: I4cacb10bac419633ca0c14f244f9903f7f517b49
Note that some GPRS and SGs related coding is done here in libosmocore and
hence currently remains using the old implementation (see previous version of
this patch: Ic3f969e739654c1e8c387aedeeba5cce07fe2307).
New API functions provide properly size-checking implementations of:
- decoding a raw MI from a bunch of MI octets;
- locating and decoding MI from a full 3GPP TS 24.008 Complete Layer 3 msgb;
- encoding to a buffer;
- encoding to the end of a msgb.
Other than the old gsm48_generate_mid(), omit a TLV tag and length from
encoding. Many callers manually stripped the tag and value after calling
gsm48_generate_mid(). The aim is to leave writing a TL to the caller entirely,
especially since some callers need to use a TvL, i.e. support a variable-size
length of 8 or 16 bit.
New validity checks so far not implemented anywhere else:
- stricter validation of number of digits of IMSI, IMEI, IMEI-SV MI.
- stricter on filler nibbles to be 0xf.
As a result, applications using osmo_mobile_identity will be stricter in
rejecting coding mistakes (some of which we currently have in our test suites,
and which we'll need to fix).
Rationale:
While implementing osmo-bsc's MSC pooling feature in osmo-bsc, this API will be
used to reduce the number of times a Mobile Identity is extracted from a raw
RSL message.
Extracting the Mobile Identity from messages has numerous duplicate
implementations across our code with various levels of specialization.
https://xkcd.com/927/
To name a few:
- libosmocore: gsm48_mi_to_string(), osmo_mi_name_buf()
- osmo-bsc: extract_sub()
- osmo-msc: mm_rx_loc_upd_req(), cm_serv_reuse_conn(), gsm48_rx_mm_serv_req(),
vlr_proc_acc_req()
We have existing functions to produce a human readable string from a Mobile
Identity, more or less awkward:
- gsm48_mi_to_string() decodes a TMSI as a decimal number. These days we use
hexadecimal TMSI everywhere.
- osmo_mi_name_buf() decodes the BCD digits from a raw MI every time, so we'd
need to pass around the raw message bytes. Also, osmo_mi_name_buf() has the
wrong signature, it should return a length like snprintf().
- osmo-bsc's extract_sub() first uses gsm48_mi_to_string() which encodes the
raw uint32_t TMSI to a string, and then calls strtoul() via
tmsi_from_string() to code those back to a raw uint32_t.
Each of the above implementations employ their own size overflow checks, each
invoke osmo_bcd2str() and implement their own TMSI osmo_load32be() handling.
Too much code dup, let's hope that each and every one is correct.
In osmo-bsc, I am now implementing MSC pooling, and need to extract NRI bits
from a TMSI Mobile Identity. Since none of the above functions are general
enough to be re-used, I found myself again copy-pasting Mobile Identity code:
locating the MI in a 24.008 message with proper size checks, decoding MI
octets.
This time I would like it to become a generally re-usable API.
This patch was first merged as Ic3f969e739654c1e8c387aedeeba5cce07fe2307 and
caused test fallout, because it re-implemented old API with the new stricter
decoding. In this patch version, old API remains 1:1 unchanged to avoid such
fallout. Applications will soon switch to the new osmo_mobile_identity API and
become stricter on MI coding when that happens, not implicitly by a new
libosmocore version.
Change-Id: If4f7be606e54cfa1c59084cf169785b1cbda5cf5
|
|
This reverts commit d1ceca9d48eb3d8b212f386a1ebb35d8fc612297, as it
introduces regressions in both osmo-msc and osmo-nitb which have been
causing failing builds for several days now.
Change-Id: I4bd958d0cd2ab4b0c4725e6d114f4404d725fcf7
|
|
Implement better API around 3GPP TS 24.008 Mobile Identity coding.
struct osmo_mobile_identity is a decoded representation of the raw Mobile
Identity, with a string representation as well as dedicated raw uint32_t TMSI.
The aim is to remove all uncertainty about decoded buffer sizes / data types.
I have patches ready for all osmo programs, completely replacing the Mobile
Identity coding with this new API. Hence deprecate the old MI API.
New API functions provide properly size-checking implementations of:
- decoding a raw MI from a bunch of MI octets;
- locating and decoding MI from a full 3GPP TS 24.008 Complete Layer 3 msgb;
- encoding to a buffer;
- encoding to the end of a msgb.
Other than the old gsm48_generate_mid(), omit a TLV tag and length from
encoding. Many callers manually stripped the tag and value after calling
gsm48_generate_mid(). The aim is to leave writing a TL to the caller entirely,
especially since some callers need to use a TvL, i.e. support a variable-size
length of 8 or 16 bit.
New validity checks so far not implemented anywhere else:
- stricter validation of number of digits of IMSI, IMEI, IMEI-SV MI.
- stricter on filler nibbles to be 0xf.
Rationale:
While implementing osmo-bsc's MSC pooling feature in osmo-bsc, this API will be
used to reduce the number of times a Mobile Identity is extracted from a raw
RSL message.
Extracting the Mobile Identity from messages has numerous duplicate
implementations across our code with various levels of specialization.
https://xkcd.com/927/
To name a few:
- libosmocore: gsm48_mi_to_string(), osmo_mi_name_buf()
- osmo-bsc: extract_sub()
- osmo-msc: mm_rx_loc_upd_req(), cm_serv_reuse_conn(), gsm48_rx_mm_serv_req(),
vlr_proc_acc_req()
We have existing functions to produce a human readable string from a Mobile
Identity, more or less awkward:
- gsm48_mi_to_string() decodes a TMSI as a decimal number. These days we use
hexadecimal TMSI everywhere.
- osmo_mi_name_buf() decodes the BCD digits from a raw MI every time, so we'd
need to pass around the raw message bytes. Also, osmo_mi_name_buf() has the
wrong signature, it should return a length like snprintf().
- osmo-bsc's extract_sub() first uses gsm48_mi_to_string() which encodes the
raw uint32_t TMSI to a string, and then calls strtoul() via
tmsi_from_string() to code those back to a raw uint32_t.
Each of the above implementations employ their own size overflow checks, each
invoke osmo_bcd2str() and implement their own TMSI osmo_load32be() handling.
Too much code dup, let's hope that each and every one is correct.
In osmo-bsc, I am now implementing MSC pooling, and need to extract NRI bits
from a TMSI Mobile Identity. Since none of the above functions are general
enough to be re-used, I found myself again copy-pasting Mobile Identity code:
locating the MI in a 24.008 message with proper size checks, decoding MI
octets.
This time I would like it to become a generally re-usable API.
Change-Id: Ic3f969e739654c1e8c387aedeeba5cce07fe2307
|
|
Provide string escaping that
- returns the required buffer size, so it can be used with OSMO_STRBUF_APPEND().
- uses C compatible string constant escaping sequences.
This is intended as a replacement for all previous osmo_escape_str* and
osmo_quote_str* API. It pains me that I didn't get them right the first nor the
second time:
- The buffer functions do not return the chars needed, which is required for
allocating sufficient memory in the *_c versions of the functions.
- Because of that, these functions are accurately usable for
OSMO_STRBUF_APPEND(), producing truncated strings, for example when dumping a
GSUP message.
- They do not use the C equivalent string constant escaping: for some reason I
thought "\15" would be valid, but it should be "\x0f".
If I could, I would completely drop those mislead implementations ... but
backwards compat prohibits that.
A previous patch already provided internal static functions that accurately
return the required buffer size. Enhance these to also support C compatible
string escaping, and use them as implementation of the new functions:
osmo_escape_cstr_buf()
osmo_escape_cstr_c()
osmo_quote_cstr_buf()
osmo_quote_cstr_c()
In the tests for these, also test C string equivalence.
Naming: from API versions, it would be kind of logical to call them
osmo_escape_str_buf3() and osmo_escape_str_c2(). Since these anyway return a
different escaping, it makes sense to me to have distinct names instead.
Quasi missing are variants of the non-C-compatible weird legacy escaping that
return the required buffer size, but I refrain from adding those, because we
have enough API cruft as it is. Just always use these new cstr variants.
Change-Id: I3dfb892036e01000033dd8e7e4a6a0c32a3caa9b
|
|
When finding a char in a string, I want to be able to limit the search area by
size, not only by nul terminator.
Change-Id: I48f8ace9f51f8a06796648883afcabe3b4e8b537
|
|
A couple of times recently I've needed to copy out a substring to a buffer with
limited size. Use of strncpy() or osmo_strlcpy() are nontrivial here.
I wanted to have a dedicated function.
After I wrote that function with a test, I noticed that I had already
implemented the same thing a while ago, as osmo_print_n() :P
So here is just the test.
Change-Id: Ia716abdc1f58af6065b84f4f567388a32a7b39fc
|
|
Provide a common implementation for foo_name_c() functions that base on
foo_name_buf() functions.
char *foo_name_c(void *ctx, example_t arg)
{
OSMO_NAME_C_IMPL(ctx, 64, "ERROR", foo_name_buf, arg)
}
Rationale: the most efficient way of composing strings that have optional parts
or require loops for composition is by writing to a ready char[], and this in
turn is easiest done by using OSMO_STRBUF_* API. Using such a basic name string
implementation which typically returns a length, I often want a more convenient
version that returns a char*, which can just be inlined in a "%s" string format
-- crucially: skipping string composition when inlined in a LOGP(). This common
implementation allows saving code dup, only the function signature is needed.
Why not include the function signature in the macro? The two sets of varargs
(1: signature args, 2: function call args) are hard to do. Also, having an
explicit signature is good for readability and code grepping / ctags.
Upcoming uses: in libosmocore in the mslookup (D-GSM) implementation
(osmo_mslookup_result_name_c()), and in osmo_msc's codec negotiation
implementation (sdp_audio_codecs_name_c(), sdp_msg_name_c(), ...).
I54b6c0810f181259da307078977d9ef3d90458c9 (libosmocore)
If3ce23cd5bab15e2ab4c52ef3e4c75979dffe931 (osmo-msc)
Change-Id: Ida5ba8d9640ea641aafef0236800f6d489d3d322
|
|
To be able to append an escaped or quoted string using
OSMO_STRBUF_APPEND_NOLEN(), the function signature must have the buf and len as
first args, like most other *_buf() functions.
Add osmo_escape_str_buf2() and osmo_quote_str_buf2() to match this signature.
A recent patch [1] has changed the return value of osmo_escape_str_buf() to
char*, removing the const. However, the functions may return const strings,
hence re-add the const. The new signatures always return the non-const buffer.
To avoid code duplication, implement osmo_quote_str_buf() and
osmo_escape_str_buf() by calling the new functions.
I decided to allow slight changes to the behavior for current osmo_escape_str()
and osmo_escape_str_buf(), because impact on callers is minimal:
(1) The new implementation uses OSMO_STRBUF_*, and in consequence
osmo_quote_str() no longer prints an ending double quote after truncated
strings; Before, a truncated output was, sic:
"this string is trunca"
and now this becomes, sic:
"this string is truncat
I decided to not keep the old behavior because it is questionable to begin
with. It looks like the string actually ended at the truncation boundary
instead of the reason being not enough space in the output buffer.
(2) The new osmo_escape_str_buf2() function obviously cannot pass-thru an
unchanged char* if no escaping was needed. Sacrifice this tiny optimization
feature to avoid code duplication:
- it is an unnoticeable optimization,
- the caller anyway always passes a string buffer,
- the feature caused handling strings and buffers differently depending on
their content (i.e. code that usually writes out strings in full length
"suddenly" truncates because a non-printable character is contained, etc.)
I considered adding a skip_if_unescaped flag to the osmo_quote_str_buf2()
function signature, but in the end decided that the API clutter is not worth
having for all the above reasons.
Adjust tests to accomodate above changes.
[1] 4a62eda225ab7f3c9556990c81a6fc5e19b5eec8
Ibf85f79e93244f53b2684ff6f1095c5b41203e05
Change-Id: Id748b906b0083b1f1887f2be7a53cae705a8a9ae
|
|
In OSMO_STRBUF_APPEND, use local variable names that are less likely to shadow
other local variables: prefix with _sb_.
In OSMO_STRBUF_APPEND, add a check to add to .pos only if it is not NULL.
Add OSMO_STRBUF_APPEND_NOLEN(), which works for function signatures that don't
return a length. This is useful for any osmo_*_buf() string writing functions,
so that these write directly to the strbuf.
Change-Id: I108cadf72deb3a3bcab9a07e50572d9da1ab0359
|
|
Move from a static implementation in tdef_vty.c to utils.c, I also want to use
this in osmo-msc.
The point is that the telnet VTY allows unambiguous partly matches of keyword
args. For example, if I have a command definition of:
compare (apples|oranges)
then it is perfectly legal as for the vty parser to write only
compare app
One could expect the VTY to then pass the unambiguous match of "apples" to the
parsing function, but that is not the case.
Hence a VTY function implementation is faced with parsing a keyword of "app"
instead of the expected "apples".
This is actually a very widespread bug in our VTY implementations, which assume
that exactly one full keyword will always be found. I am now writing new
commands in a way that are able to manage only the starts of keywords.
Arguably, strstr(a, b) == a does the same thing, but it searches the entire
string unnecessarily.
Change-Id: Ib2ffb0e9a870dd52e081c7e66d8818057d159513
|
|
osmo_escape_str_buf() used to have the somewhat odd semantics that
if no escaping was needed, it would return the original pointer without
making any copy to the output buffer. While this seems like an elegant
optimization, it is a very strange behavior and it works differently
than all of our other *_buf() functions. Let's unify the API and
turn osmo_escape_str_buf() into a strlcpy() if no escaping is needed.
Change-Id: I3a02bdb27008a73101c2db41ac04248960ed4064
|
|
We are using macros like this or different workarounds in libmsc. In the course
of implementing inter-MSC handover, I am encountering yet another such
situation of appending multiple strings to a limited char buffer. Standardize.
Add a unit test to utils_test.c.
Change-Id: I2497514e26c5e7a5d88985fc7e58343be1a027b2
|
|
Add osmo_hexdump_buf() as an all-purpose hexdump function, which all other
osmo_hexdump_*() implementations now call. It absorbs the static
_osmo_hexdump(). Add tests for osmo_hexdump_buf().
Rationale: recently during patch review, a situation came up where two hexdumps
in a single printf would have been useful. Now I've faced a similar situation
again, in ongoing development. So I decided it is time to provide this API.
The traditional osmo_hexdump() API returns a non-const char*, which should
probably have been a const instead. Particularly this new function may return a
string constant "" if the buf is NULL or empty, so return const char*. That is
why the older implementations calling osmo_hexdump_buf() separately return the
buffer instead of the const return value directly.
Change-Id: I590595567b218b24e53c9eb1fd8736c0324d371d
|
|
Add a standalone bcd-to-string conversion function with generic parameters.
Add a regression test in utils_test.c.
So far there is no single universal implementation that converts a BCD to a
string. I could only find gsm48_mi_to_string(), which also interprets
surrounding bytes, MI type and TMSI as non-BCD value.
The idea is to use this function from gsm48_mi_to_string() and similar
implementations in subsequent commits.
Root cause: in osmo-msc, I want to have an alternative MI-to-string function
for composing an FSM name, which needs the BCD part of gsm48_mi_to_string() but
not the TMSI part.
Change-Id: I86b09d37ceef33331c1a56046a5443127d6c6be0
|
|
We already have osmo_str2lower() and osmo_str2upper(), but these lack:
* proper destination buffer bounds checking,
* ability to call directly as printf() argument.
Deprecate osmo_str2upper() and osmo_str2lower() because of missing bounds
checking.
Introduce osmo_str_tolower_buf(), osmo_str_toupper_buf() to provide
bounds-safe conversion, also able to safely convert a buffer in-place.
Introduce osmo_str_tolower(), osmo_str_toupper() that call the above _buf()
equivalents using a static buffer[128] and returning the resulting string
directly, convenient for direct printing. Possibly truncated but always safe.
Add unit tests to utils_test.c.
Replace all libosmocore uses of now deprecated osmo_str2lower().
Naming: the ctype.h API is called tolower() and toupper(), so just prepend
'osmo_str_' and don't separate 'to_lower'.
Change-Id: Ib0ee1206b9f31d7ba25c31f8008119ac55440797
|
|
In the past, the function ipa_ccm_idtag_parse() was used to parse
the payload of IPA CCM ID RESP packets. However, the function was
based on a possible misunderstanding of the message encoding, and
callers actually counted the first (upper) length nibble as part
of the header and passed a pointer to the second
(lower) length nibble of the first TLV into this function. As such,
it was unfixable, and had to be replaced with a new function called
ipa_ccm_id_resp_parse(). At the same time, we also add
ipa_ccm_id_get_parse() to parse the slightly different format of
the IPA CCM ID GET payload.
We can never be 100% sure what is "correct", as our understanding
of the protocol is entirely based on protocol analysis, without any
official documentation available.
This patch also introduces unit test coverage for both of the new
functions.
Revert "ipa: Add libosmogsm.map entry for ipa_ccm_idtag_parse_off"
This reverts commit 7f31c90b80c08fbfe2d84d70d397402fdb38b94c.
Revert "ipa: Properly parse LV stream of a ID_GET request"
This reverts commit f558ed4bb9c0f00997b8f97c2b251a574c1a64c4.
It introduced a function/behavior that was not originally intended:
The parse of IPA CCM ID GET (8bit length followed by 1 byte tag
and variable-length payload) instead of the IPA CCM ID RESP (16bit
length followed by 1 byte tag and variable-length payload).
Change-Id: I1834d90fbcdbfcb05f5b8cfe39bfe9543737ef8f
|
|
Change-Id: I1ebeba2067549e0dd1541fa84715d44321ff3b43
|
|
This came from osmo-bsc refactoring patch I82e3f918295daa83274a4cf803f046979f284366
https://gerrit.osmocom.org/#/c/osmo-bsc/+/9671/6/src/osmo-bsc/gsm_data.c@1708
Add regression test in utils_test.c.
Change-Id: I1f2918418c38918c5ac70acaa51a47adfca12b5e
|
|
Recent OS#3407 shows that we should verify stderr to catch sanitizer failures.
(They might not always be ignorable like that one.)
Change-Id: Ic9e437a1cc96ae081e0fd6a9b6e3156987e14c0c
|
|
Multiplying the uint16_t x by itself seems to default to be calculated in
int32_t range, while it obviously needs uint32_t. This causes sporadic
sanitizer barfs:
Testing integer square-root
utils_test.c:445:18: runtime error: signed integer overflow: 60369 * 60369 cannot be represented in type 'int'
The final result is still correct, because it is in fact interpreted as uint32_t.
Cast to uint32_t to make sure the sanitizer doesn't complain.
Related: OS#3407
Change-Id: I83c14e38deaa466d977ee43c9420534ed90f090d
|
|
Change-Id: I2b96db6e037e72e92317fec874877e473a1cf909
|
|
Rationale: with osmo_escape_str(), you get the escaped contents of the string,
but not so graceful handling of NULL strings. The caller needs to quote it, and
for NULL strings not quote it.
osmo_quote_str() is like osmo_escape_str() but always quotes a non-NULL string,
and for a NULL string returns a literal NULL, i.e. it should (tm) give the
exact C representation of a string.
That's useful in testing, to show exactly what char* situation we have, without
jumping through hoops like
if (str)
printf("\"%s\"", osmo_escape_str(str, -1));
else
printf("NULL");
Copy the unit test for osmo_escape_str() and adjust. To indicate that the
double quotes are returned by osmo_quote_str(), use single quotes in the test
printf()s.
I considered allowing to pick the quoting characters by further arguments, but
that complicates things: we'd need to escape the quoting characters. Just
hardcode double quotes like C.
Change-Id: I6f1b3709b32c23fc52f70ad9ecc9439c62b02a12
|
|
It seems with default flags in_buf was being memzeroed by the compiler.
When compiling with -O0, that's not the case anymore and printf prints
after first 16 bytes, printing extra garbage which doesn't match the
expected output.
Change-Id: I736c1e4d625f647d3bb794fa717256e9dbf36e87
|
|
To report invalid characters in identifiers, it is desirable to escape any
weird characters. Otherwise we might print stray newlines or control characters
in the log output.
ctrl_test.c already uses a print_escaped() function, which will be replaced by
osmo_escape_str() in a subsequent patch.
control_cmd.c will use osmo_escape_str() to log invalid identifiers.
Change-Id: Ic685eb63dead3967d01aaa4f1e9899e5461ca49a
|
|
Sounds stupid, but we actually didn't support hex nibbles in one of
the two directions of the conversion, so let's make sure we test for
this.
Change-Id: I8445da54cc4f9b1cd64f286c2b238f4f7c87accb
|
|
Will be used by OsmoHLR to validate VTY and CTRL input.
Change-Id: Idf75946eb0a84e145adad13fc7c78bb7a267aa0a
|
|
This is particularly useful for hex dumps containing spaces found in a log
(e.g. osmo-nitb authentication rand token), which can now be passed in quotes
to osmo-auc-gen without having to edit the spaces away.
Change-Id: Ib7af07f674a2d26c8569acdee98835fb3e626c45
|
|
Change-Id: Ic95ab00b57d54905a235109561c00419161cf4bc
|
|
For some reason the structure is closer to be a LV (length
and value). The value is actually a tag but it is counted
inside the length. Introduce an overload of the parse function
to provide an offset for the length. This will be taken from
the returned length.
|
|
In the osmo-bts and libosmo-abis code the hexdump routine is used
for every incoming/outgoing packet (including voice frames) and the
usage of snprintf showed up inside profiles.
There is a semantic change when more than 4096 characters are used.
The code will now truncate at byte boundaries (and not nibbles).
Code:
static const int lengths[] = { 23, 1000, 52 };
char buf[4096];
int i;
for (i = 0; i < 30000; ++i)
char *res = osmo_hexdump(buf, lengths[i & 3]);
Results:
before: after:
real 0m3.233s real 0m0.085s
user 0m3.212s user 0m0.084s
sys 0m0.000s sys 0m0.000s
|
|
This code makes a simple dump and tests for the corner case
|
