aboutsummaryrefslogtreecommitdiff
path: root/doc
diff options
context:
space:
mode:
authorMike Pall <mike>2021-03-25 02:21:31 +0100
committerMike Pall <mike>2021-03-25 02:21:31 +0100
commit4c6b669c419f313306b9e6ee43be4ad5f6d73ec6 (patch)
tree4547f52836e186e94b68b331ea73d6b0086cb415 /doc
parent836fb5bbd3a0d48cf3e4de70535925a85aea835f (diff)
downloadluajit-4c6b669c419f313306b9e6ee43be4ad5f6d73ec6.tar.gz
luajit-4c6b669c419f313306b9e6ee43be4ad5f6d73ec6.tar.bz2
luajit-4c6b669c419f313306b9e6ee43be4ad5f6d73ec6.zip
String buffers, part 1: object serialization.
Sponsored by fmad.io.
Diffstat (limited to 'doc')
-rw-r--r--doc/contact.html2
-rw-r--r--doc/ext_buffer.html275
-rw-r--r--doc/ext_c_api.html2
-rw-r--r--doc/ext_ffi.html2
-rw-r--r--doc/ext_ffi_api.html2
-rw-r--r--doc/ext_ffi_semantics.html2
-rw-r--r--doc/ext_ffi_tutorial.html2
-rw-r--r--doc/ext_jit.html2
-rw-r--r--doc/ext_profiler.html2
-rw-r--r--doc/extensions.html2
-rw-r--r--doc/faq.html2
-rw-r--r--doc/install.html2
-rw-r--r--doc/luajit.html2
-rw-r--r--doc/running.html2
-rw-r--r--doc/status.html2
15 files changed, 303 insertions, 0 deletions
diff --git a/doc/contact.html b/doc/contact.html
index b7980091..c253a08b 100644
--- a/doc/contact.html
+++ b/doc/contact.html
@@ -37,6 +37,8 @@
37<a href="ext_ffi_semantics.html">FFI Semantics</a> 37<a href="ext_ffi_semantics.html">FFI Semantics</a>
38</li></ul> 38</li></ul>
39</li><li> 39</li><li>
40<a href="ext_buffer.html">String Buffers</a>
41</li><li>
40<a href="ext_jit.html">jit.* Library</a> 42<a href="ext_jit.html">jit.* Library</a>
41</li><li> 43</li><li>
42<a href="ext_c_api.html">Lua/C API</a> 44<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_buffer.html b/doc/ext_buffer.html
new file mode 100644
index 00000000..455c298d
--- /dev/null
+++ b/doc/ext_buffer.html
@@ -0,0 +1,275 @@
1<!DOCTYPE html>
2<html>
3<head>
4<title>String Buffers</title>
5<meta charset="utf-8">
6<meta name="Copyright" content="Copyright (C) 2005-2021">
7<meta name="Language" content="en">
8<link rel="stylesheet" type="text/css" href="bluequad.css" media="screen">
9<link rel="stylesheet" type="text/css" href="bluequad-print.css" media="print">
10</head>
11<body>
12<div id="site">
13<a href="https://luajit.org"><span>Lua<span id="logo">JIT</span></span></a>
14</div>
15<div id="head">
16<h1>String Buffers</h1>
17</div>
18<div id="nav">
19<ul><li>
20<a href="luajit.html">LuaJIT</a>
21<ul><li>
22<a href="https://luajit.org/download.html">Download <span class="ext">&raquo;</span></a>
23</li><li>
24<a href="install.html">Installation</a>
25</li><li>
26<a href="running.html">Running</a>
27</li></ul>
28</li><li>
29<a href="extensions.html">Extensions</a>
30<ul><li>
31<a href="ext_ffi.html">FFI Library</a>
32<ul><li>
33<a href="ext_ffi_tutorial.html">FFI Tutorial</a>
34</li><li>
35<a href="ext_ffi_api.html">ffi.* API</a>
36</li><li>
37<a href="ext_ffi_semantics.html">FFI Semantics</a>
38</li></ul>
39</li><li>
40<a class="current" href="ext_buffer.html">String Buffers</a>
41</li><li>
42<a href="ext_jit.html">jit.* Library</a>
43</li><li>
44<a href="ext_c_api.html">Lua/C API</a>
45</li><li>
46<a href="ext_profiler.html">Profiler</a>
47</li></ul>
48</li><li>
49<a href="status.html">Status</a>
50</li><li>
51<a href="faq.html">FAQ</a>
52</li><li>
53<a href="http://wiki.luajit.org/">Wiki <span class="ext">&raquo;</span></a>
54</li><li>
55<a href="https://luajit.org/list.html">Mailing List <span class="ext">&raquo;</span></a>
56</li></ul>
57</div>
58<div id="main">
59<p>
60
61The string buffer library allows <b>high-performance manipulation of
62string-like data</b>.
63
64</p>
65<p>
66
67Unlike Lua strings, which are constants, string buffers are
68<b>mutable</b> sequences of 8-bit (binary-transparent) characters. Data
69can be stored, formatted and encoded into a string buffer and later
70converted, decoded or extracted.
71
72</p>
73<p>
74
75The convenient string buffer API simplifies common string manipulation
76tasks, that would otherwise require creating many intermediate strings.
77String buffers improve performance by eliminating redundant memory
78copies, object creation, string interning and garbage collection
79overhead. In conjunction with the FFI library, they allow zero-copy
80operations.
81
82</p>
83
84<h2 id="load">Using the String Buffer Library</h2>
85<p>
86The string buffer library is built into LuaJIT by default, but it's not
87loaded by default. Add this to the start of every Lua file that needs
88one of its functions:
89</p>
90<pre class="code">
91local buffer = require("string.buffer")
92</pre>
93
94<h2 id="wip" style="color:#ff0000">Work in Progress</h2>
95
96<p>
97
98<b style="color:#ff0000">This library is a work in progress. More
99functions will be added soon.</b>
100
101</p>
102
103<h2 id="serialize">Serialization of Lua Objects</h2>
104<p>
105
106The following functions and methods allow <b>high-speed serialization</b>
107(encoding) of a Lua object into a string and decoding it back to a Lua
108object. This allows convenient storage and transport of <b>structured
109data</b>.
110
111</p>
112<p>
113
114The encoded data is in an <a href="#serialize_format">internal binary
115format</a>. The data can be stored in files, binary-transparent
116databases or transmitted to other LuaJIT instances across threads,
117processes or networks.
118
119</p>
120<p>
121
122Encoding speed can reach up to 1 Gigabyte/second on a modern desktop- or
123server-class system, even when serializing many small objects. Decoding
124speed is mostly constrained by object creation cost.
125
126</p>
127<p>
128
129The serializer handles most Lua types, common FFI number types and
130nested structures. Functions, thread objects, other FFI cdata, full
131userdata and associated metatables cannot be serialized (yet).
132
133</p>
134<p>
135
136The encoder serializes nested structures as trees. Multiple references
137to a single object will be stored separately and create distinct objects
138after decoding. Circular references cause an error.
139
140
141</p>
142
143<h3 id="buffer_encode"><tt>str = buffer.encode(obj)</tt></h3>
144<p>
145
146Serializes (encodes) the Lua object <tt>obj</tt> into the string
147<tt>str</tt>.
148
149</p>
150<p>
151
152<tt>obj</tt> can be any of the supported Lua types &mdash; it doesn't
153need to be a Lua table.
154
155</p>
156<p>
157
158This function may throw an error when attempting to serialize
159unsupported object types, circular references or deeply nested tables.
160
161</p>
162
163<h3 id="buffer_decode"><tt>obj = buffer.decode(str)</tt></h3>
164<p>
165
166De-serializes (decodes) the string <tt>str</tt> into the Lua object
167<tt>obj</tt>.
168
169</p>
170<p>
171
172The returned object may be any of the supported Lua types &mdash;
173even <tt>nil</tt>.
174
175</p>
176<p>
177
178This function may throw an error when fed with malformed or incomplete
179encoded data. The standalone function throws when there's left-over data
180after decoding a single top-level object.
181
182</p>
183
184<h2 id="serialize_format">Serialization Format Specification</h2>
185<p>
186
187This serialization format is designed for <b>internal use</b> by LuaJIT
188applications. Serialized data is upwards-compatible and portable across
189all supported LuaJIT platforms.
190
191</p>
192<p>
193
194It's an <b>8-bit binary format</b> and not human-readable. It uses e.g.
195embedded zeroes and stores embedded Lua string objects unmodified, which
196are 8-bit-clean, too. Encoded data can be safely concatenated for
197streaming and later decoded one top-level object at a time.
198
199</p>
200<p>
201
202The encoding is reasonably compact, but tuned for maximum performance,
203not for minimum space usage. It compresses well with any of the common
204byte-oriented data compression algorithms.
205
206</p>
207<p>
208
209Although documented here for reference, this format is explicitly
210<b>not</b> intended to be a 'public standard' for structured data
211interchange across computer languages (like JSON or MessagePack). Please
212do not use it as such.
213
214</p>
215<p>
216
217The specification is given below as a context-free grammar with a
218top-level <tt>object</tt> as the starting point. Alternatives are
219separated by the <tt>|</tt> symbol and <tt>*</tt> indicates repeats.
220Grouping is implicit or indicated by <tt>{…}</tt>. Terminals are
221either plain hex numbers, encoded as bytes, or have a <tt>.format</tt>
222suffix.
223
224</p>
225<pre>
226object → nil | false | true
227 | null | lightud32 | lightud64
228 | int | num | tab
229 | int64 | uint64 | complex
230 | string
231
232nil → 0x00
233false → 0x01
234true → 0x02
235
236null → 0x03 // NULL lightuserdata
237lightud32 → 0x04 data.I // 32 bit lightuserdata
238lightud64 → 0x05 data.L // 64 bit lightuserdata
239
240int → 0x06 int.I // int32_t
241num → 0x07 double.L
242
243tab → 0x08 // Empty table
244 | 0x09 h.U h*{object object} // Key/value hash
245 | 0x0a a.U a*object // 0-based array
246 | 0x0b a.U a*object h.U h*{object object} // Mixed
247 | 0x0c a.U (a-1)*object // 1-based array
248 | 0x0d a.U (a-1)*object h.U h*{object object} // Mixed
249
250int64 → 0x10 int.L // FFI int64_t
251uint64 → 0x11 uint.L // FFI uint64_t
252complex → 0x12 re.L im.L // FFI complex
253
254string → (0x20+len).U len*char.B
255
256.B = 8 bit
257.I = 32 bit little-endian
258.L = 64 bit little-endian
259.U = prefix-encoded 32 bit unsigned number n:
260 0x00..0xdf → n.B
261 0xe0..0x1fdf → (0xe0|(((n-0xe0)>>8)&0x1f)).B ((n-0xe0)&0xff).B
262 0x1fe0.. → 0xff n.I
263</pre>
264<br class="flush">
265</div>
266<div id="foot">
267<hr class="hide">
268Copyright &copy; 2005-2021
269<span class="noprint">
270&middot;
271<a href="contact.html">Contact</a>
272</span>
273</div>
274</body>
275</html>
diff --git a/doc/ext_c_api.html b/doc/ext_c_api.html
index 6079e5ac..9f1ad212 100644
--- a/doc/ext_c_api.html
+++ b/doc/ext_c_api.html
@@ -37,6 +37,8 @@
37<a href="ext_ffi_semantics.html">FFI Semantics</a> 37<a href="ext_ffi_semantics.html">FFI Semantics</a>
38</li></ul> 38</li></ul>
39</li><li> 39</li><li>
40<a href="ext_buffer.html">String Buffers</a>
41</li><li>
40<a href="ext_jit.html">jit.* Library</a> 42<a href="ext_jit.html">jit.* Library</a>
41</li><li> 43</li><li>
42<a class="current" href="ext_c_api.html">Lua/C API</a> 44<a class="current" href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_ffi.html b/doc/ext_ffi.html
index 13b75bda..b934dc78 100644
--- a/doc/ext_ffi.html
+++ b/doc/ext_ffi.html
@@ -37,6 +37,8 @@
37<a href="ext_ffi_semantics.html">FFI Semantics</a> 37<a href="ext_ffi_semantics.html">FFI Semantics</a>
38</li></ul> 38</li></ul>
39</li><li> 39</li><li>
40<a href="ext_buffer.html">String Buffers</a>
41</li><li>
40<a href="ext_jit.html">jit.* Library</a> 42<a href="ext_jit.html">jit.* Library</a>
41</li><li> 43</li><li>
42<a href="ext_c_api.html">Lua/C API</a> 44<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_ffi_api.html b/doc/ext_ffi_api.html
index b7ace808..061cc42a 100644
--- a/doc/ext_ffi_api.html
+++ b/doc/ext_ffi_api.html
@@ -42,6 +42,8 @@ td.abiparam { font-weight: bold; width: 6em; }
42<a href="ext_ffi_semantics.html">FFI Semantics</a> 42<a href="ext_ffi_semantics.html">FFI Semantics</a>
43</li></ul> 43</li></ul>
44</li><li> 44</li><li>
45<a href="ext_buffer.html">String Buffers</a>
46</li><li>
45<a href="ext_jit.html">jit.* Library</a> 47<a href="ext_jit.html">jit.* Library</a>
46</li><li> 48</li><li>
47<a href="ext_c_api.html">Lua/C API</a> 49<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_ffi_semantics.html b/doc/ext_ffi_semantics.html
index 904ee51d..fef39c32 100644
--- a/doc/ext_ffi_semantics.html
+++ b/doc/ext_ffi_semantics.html
@@ -42,6 +42,8 @@ td.convop { font-style: italic; width: 40%; }
42<a class="current" href="ext_ffi_semantics.html">FFI Semantics</a> 42<a class="current" href="ext_ffi_semantics.html">FFI Semantics</a>
43</li></ul> 43</li></ul>
44</li><li> 44</li><li>
45<a href="ext_buffer.html">String Buffers</a>
46</li><li>
45<a href="ext_jit.html">jit.* Library</a> 47<a href="ext_jit.html">jit.* Library</a>
46</li><li> 48</li><li>
47<a href="ext_c_api.html">Lua/C API</a> 49<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_ffi_tutorial.html b/doc/ext_ffi_tutorial.html
index 8ed61364..ca71be4d 100644
--- a/doc/ext_ffi_tutorial.html
+++ b/doc/ext_ffi_tutorial.html
@@ -44,6 +44,8 @@ td.idiomlua b { font-weight: normal; color: #2142bf; }
44<a href="ext_ffi_semantics.html">FFI Semantics</a> 44<a href="ext_ffi_semantics.html">FFI Semantics</a>
45</li></ul> 45</li></ul>
46</li><li> 46</li><li>
47<a href="ext_buffer.html">String Buffers</a>
48</li><li>
47<a href="ext_jit.html">jit.* Library</a> 49<a href="ext_jit.html">jit.* Library</a>
48</li><li> 50</li><li>
49<a href="ext_c_api.html">Lua/C API</a> 51<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_jit.html b/doc/ext_jit.html
index 84302fa0..6dd54c70 100644
--- a/doc/ext_jit.html
+++ b/doc/ext_jit.html
@@ -37,6 +37,8 @@
37<a href="ext_ffi_semantics.html">FFI Semantics</a> 37<a href="ext_ffi_semantics.html">FFI Semantics</a>
38</li></ul> 38</li></ul>
39</li><li> 39</li><li>
40<a href="ext_buffer.html">String Buffers</a>
41</li><li>
40<a class="current" href="ext_jit.html">jit.* Library</a> 42<a class="current" href="ext_jit.html">jit.* Library</a>
41</li><li> 43</li><li>
42<a href="ext_c_api.html">Lua/C API</a> 44<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/ext_profiler.html b/doc/ext_profiler.html
index 0e8d3691..2783abdb 100644
--- a/doc/ext_profiler.html
+++ b/doc/ext_profiler.html
@@ -37,6 +37,8 @@
37<a href="ext_ffi_semantics.html">FFI Semantics</a> 37<a href="ext_ffi_semantics.html">FFI Semantics</a>
38</li></ul> 38</li></ul>
39</li><li> 39</li><li>
40<a href="ext_buffer.html">String Buffers</a>
41</li><li>
40<a href="ext_jit.html">jit.* Library</a> 42<a href="ext_jit.html">jit.* Library</a>
41</li><li> 43</li><li>
42<a href="ext_c_api.html">Lua/C API</a> 44<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/extensions.html b/doc/extensions.html
index 77cf444c..799679a3 100644
--- a/doc/extensions.html
+++ b/doc/extensions.html
@@ -54,6 +54,8 @@ td.excinterop {
54<a href="ext_ffi_semantics.html">FFI Semantics</a> 54<a href="ext_ffi_semantics.html">FFI Semantics</a>
55</li></ul> 55</li></ul>
56</li><li> 56</li><li>
57<a href="ext_buffer.html">String Buffers</a>
58</li><li>
57<a href="ext_jit.html">jit.* Library</a> 59<a href="ext_jit.html">jit.* Library</a>
58</li><li> 60</li><li>
59<a href="ext_c_api.html">Lua/C API</a> 61<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/faq.html b/doc/faq.html
index b71e6e7c..a5d744d2 100644
--- a/doc/faq.html
+++ b/doc/faq.html
@@ -40,6 +40,8 @@ dd { margin-left: 1.5em; }
40<a href="ext_ffi_semantics.html">FFI Semantics</a> 40<a href="ext_ffi_semantics.html">FFI Semantics</a>
41</li></ul> 41</li></ul>
42</li><li> 42</li><li>
43<a href="ext_buffer.html">String Buffers</a>
44</li><li>
43<a href="ext_jit.html">jit.* Library</a> 45<a href="ext_jit.html">jit.* Library</a>
44</li><li> 46</li><li>
45<a href="ext_c_api.html">Lua/C API</a> 47<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/install.html b/doc/install.html
index fab0b2ca..e4af9dde 100644
--- a/doc/install.html
+++ b/doc/install.html
@@ -65,6 +65,8 @@ td.compatno {
65<a href="ext_ffi_semantics.html">FFI Semantics</a> 65<a href="ext_ffi_semantics.html">FFI Semantics</a>
66</li></ul> 66</li></ul>
67</li><li> 67</li><li>
68<a href="ext_buffer.html">String Buffers</a>
69</li><li>
68<a href="ext_jit.html">jit.* Library</a> 70<a href="ext_jit.html">jit.* Library</a>
69</li><li> 71</li><li>
70<a href="ext_c_api.html">Lua/C API</a> 72<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/luajit.html b/doc/luajit.html
index 42c0ac83..a25267a6 100644
--- a/doc/luajit.html
+++ b/doc/luajit.html
@@ -122,6 +122,8 @@ table.feature small {
122<a href="ext_ffi_semantics.html">FFI Semantics</a> 122<a href="ext_ffi_semantics.html">FFI Semantics</a>
123</li></ul> 123</li></ul>
124</li><li> 124</li><li>
125<a href="ext_buffer.html">String Buffers</a>
126</li><li>
125<a href="ext_jit.html">jit.* Library</a> 127<a href="ext_jit.html">jit.* Library</a>
126</li><li> 128</li><li>
127<a href="ext_c_api.html">Lua/C API</a> 129<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/running.html b/doc/running.html
index ae4911d5..b55b8439 100644
--- a/doc/running.html
+++ b/doc/running.html
@@ -59,6 +59,8 @@ td.param_default {
59<a href="ext_ffi_semantics.html">FFI Semantics</a> 59<a href="ext_ffi_semantics.html">FFI Semantics</a>
60</li></ul> 60</li></ul>
61</li><li> 61</li><li>
62<a href="ext_buffer.html">String Buffers</a>
63</li><li>
62<a href="ext_jit.html">jit.* Library</a> 64<a href="ext_jit.html">jit.* Library</a>
63</li><li> 65</li><li>
64<a href="ext_c_api.html">Lua/C API</a> 66<a href="ext_c_api.html">Lua/C API</a>
diff --git a/doc/status.html b/doc/status.html
index e1f024bf..1d3ba984 100644
--- a/doc/status.html
+++ b/doc/status.html
@@ -40,6 +40,8 @@ ul li { padding-bottom: 0.3em; }
40<a href="ext_ffi_semantics.html">FFI Semantics</a> 40<a href="ext_ffi_semantics.html">FFI Semantics</a>
41</li></ul> 41</li></ul>
42</li><li> 42</li><li>
43<a href="ext_buffer.html">String Buffers</a>
44</li><li>
43<a href="ext_jit.html">jit.* Library</a> 45<a href="ext_jit.html">jit.* Library</a>
44</li><li> 46</li><li>
45<a href="ext_c_api.html">Lua/C API</a> 47<a href="ext_c_api.html">Lua/C API</a>
generated by cgit v1.2.3-55-g6feb (git 2.39.1) at 2026-01-11 20:21:02 +0000