1FASTAVRO(1) fastavro FASTAVRO(1)
2
6 fastavro - fastavro Documentation
7 The current Python avro package is packed with features but dog slow.
9 On a test case of about 10K records, it takes about 14sec to iterate
11 over all of them. In comparison the JAVA avro SDK does it in about
12 1.9sec.
13 fastavro is less feature complete than avro, however it’s much faster.
15 It iterates over the same 10K records in 2.9sec, and if you use it with
16 PyPy it’ll do it in 1.5sec (to be fair, the JAVA benchmark is doing
17 some extra JSON encoding/decoding).
18 If the optional C extension (generated by Cython) is available, then
20 fastavro will be even faster. For the same 10K records it’ll run in
21 about 1.7sec.
22
24 # Writing
25 from fastavro import writer
26 schema = {
28 'doc': 'A weather reading.',
29 'name': 'Weather',
30 'namespace': 'test',
31 'type': 'record',
32 'fields': [
33 {'name': 'station', 'type': 'string'},
34 {'name': 'time', 'type': 'long'},
35 {'name': 'temp', 'type': 'int'},
36 ],
37 }
38 # 'records' can be an iterable (including generator)
40 records = [
41 {u'station': u'011990-99999', u'temp': 0, u'time': 1433269388},
42 {u'station': u'011990-99999', u'temp': 22, u'time': 1433270389},
43 {u'station': u'011990-99999', u'temp': -11, u'time': 1433273379},
44 {u'station': u'012650-99999', u'temp': 111, u'time': 1433275478},
45 ]
46 with open('weather.avro', 'wb') as out:
48 writer(out, schema, records)
49 # Reading
51 import fastavro
52 with open('weather.avro', 'rb') as fo:
54 reader = fastavro.reader(fo)
55 schema = reader.schema
56 for record in reader:
58 print(record)
59
61 fastavro.read
62 class reader(fo, reader_schema=None)
63 Iterator over records in an avro file.
64 Parameters
66 · fo (file-like) – Input stream
68 · reader_schema (dict, optional) – Reader schema
70 Example:
72 from fastavro import reader
74 with open('some-file.avro', 'rb') as fo:
75 avro_reader = reader(fo)
76 schema = avro_reader.schema
77 for record in avro_reader:
78 process_record(record)
79 class block_reader(fo, reader_schema=None)
81 Iterator over blocks in an avro file.
82 Parameters
84 · fo (file-like) – Input stream
86 · reader_schema (dict, optional) – Reader schema
88 Example:
90 from fastavro import block_reader
92 with open('some-file.avro', 'rb') as fo:
93 avro_reader = block_reader(fo)
94 schema = avro_reader.schema
95 for block in avro_reader:
96 process_block(block)
97 schemaless_reader(fo, writer_schema, reader_schema=None)
99 Reads a single record writen using the schemaless_writer
100 Parameters
102 · fo (file-like) – Input stream
104 · writer_schema (dict) – Schema used when calling schema‐
106 less_writer
107 · reader_schema (dict, optional) – If the schema has
109 changed since being written then the new schema can be
110 given to allow for schema migration
111 Example:
113 with open('file.avro', 'rb') as fp:
115 record = fastavro.schemaless_reader(fp, schema)
116 Note: The schemaless_reader can only read a single record.
118 is_avro(path_or_buffer)
120 Return True if path (or buffer) points to an Avro file.
121 Parameters
123 path_or_buffer (path to file or file-like object) – Path
124 to file
125 fastavro.write
127 writer(fo, schema, records, codec='null', sync_interval=16000, meta‐
128 data=None, validator=None)
129 Write records to fo (stream) according to schema
130 Parameters
132 · fo (file-like) – Output stream
134 · records (iterable) – Records to write
136 · codec (string, optional) – Compression codec, can be
138 ‘null’, ‘deflate’ or ‘snappy’ (if installed)
139 · sync_interval (int, optional) – Size of sync interval
141 · metadata (dict, optional) – Header metadata
143 · validator (None, True or a function) – Validator func‐
145 tion. If None (the default) - no validation. If True
146 then then fastavro.validation.validate will be used. If
147 it’s a function, it should have the same signature as
148 fastavro.writer.validate and raise an exeption on
149 error.
150 Example:
152 from fastavro import writer
154 schema = {
156 'doc': 'A weather reading.',
157 'name': 'Weather',
158 'namespace': 'test',
159 'type': 'record',
160 'fields': [
161 {'name': 'station', 'type': 'string'},
162 {'name': 'time', 'type': 'long'},
163 {'name': 'temp', 'type': 'int'},
164 ],
165 }
166 records = [
168 {u'station': u'011990-99999', u'temp': 0, u'time': 1433269388},
169 {u'station': u'011990-99999', u'temp': 22, u'time': 1433270389},
170 {u'station': u'011990-99999', u'temp': -11, u'time': 1433273379},
171 {u'station': u'012650-99999', u'temp': 111, u'time': 1433275478},
172 ]
173 with open('weather.avro', 'wb') as out:
175 writer(out, schema, records)
176 schemaless_writer(fo, schema, record)
178 Write a single record without the schema or header information
179 Parameters
181 · fo (file-like) – Output file
183 · schema (dict) – Schema
185 · record (dict) – Record to write
187 Example:
189 with open('file.avro', 'rb') as fp:
191 fastavro.schemaless_writer(fp, schema, record)
192 Note: The schemaless_writer can only write a single record.
194 fastavro.validation
196 validate(datum, schema, field=None, raise_errors=True)
197 Determine if a python datum is an instance of a schema.
198 Parameters
200 · datum (Any) – Data being validated
202 · schema (dict) – Schema
204 · field (str, optional) – Record field being validated
206 · raise_errors (bool, optional) – If true, errors are
208 raised for invalid data. If false, a simple True
209 (valid) or False (invalid) result is returned
210 Example:
212 from fastavro.validation import validate
214 schema = {...}
215 record = {...}
216 validate(record, schema)
217 validate_many(records, schema, raise_errors=True)
219 Validate a list of data!
220 Parameters
222 · records (iterable) – List of records to validate
224 · schema (dict) – Schema
226 · raise_errors (bool, optional) – If true, errors are
228 raised for invalid data. If false, a simple True
229 (valid) or False (invalid) result is returned
230 Example:
232 from fastavro.validation import validate_many
234 schema = {...}
235 records = [{...}, {...}, ...]
236 validate_many(records, schema)
237 fastavro command line script
239 A command line script is installed with the library that can be used to
240 dump the contents of avro file(s) to the standard output.
241 Usage:
243 usage: fastavro [-h] [--schema] [--codecs] [--version] [-p] [file [file ...]]
245 iter over avro file, emit records as JSON
247 positional arguments:
249 file file(s) to parse
250 optional arguments:
252 -h, --help show this help message and exit
253 --schema dump schema instead of records
254 --codecs print supported codecs
255 --version show program's version number and exit
256 -p, --pretty pretty print json
257 Examples
259 Read an avro file:
260 $ fastavro weather.avro
262 {"temp": 0, "station": "011990-99999", "time": -619524000000}
264 {"temp": 22, "station": "011990-99999", "time": -619506000000}
265 {"temp": -11, "station": "011990-99999", "time": -619484400000}
266 {"temp": 111, "station": "012650-99999", "time": -655531200000}
267 {"temp": 78, "station": "012650-99999", "time": -655509600000}
268 Show the schema:
270 $ fastavro --schema weather.avro
272 {
274 "type": "record",
275 "namespace": "test",
276 "doc": "A weather reading.",
277 "fields": [
278 {
279 "type": "string",
280 "name": "station"
281 },
282 {
283 "type": "long",
284 "name": "time"
285 },
286 {
287 "type": "int",
288 "name": "temp"
289 }
290 ],
291 "name": "Weather"
292 }
293 · genindex
295 · modindex
297 · search
299
301 Miki Tebeka
302
304 2012, Miki Tebeka
3050.19.8 Jul 15, 2018 FASTAVRO(1)