Squashed 'third_party/flatbuffers/' changes from acc9990ab..d6a8dbd26
d6a8dbd26 Experimental fix for failing oss-fuzz coverage build (#6259)
ed391e177 BREAKING: Rust flexbuffers serde human readable set to false (#6257)
a49531414 Update to flags in fuzzing-cmake file (#6256)
de1f0342c Remove _POSIX_C_SOURCE and _XOPEN_SOURCE definitions when compiling o… (#6205)
d0d51e2a5 flatc should support --binary --schema with optional scalar fields. (#6252)
33ab26017 Bump version of rules_go to 0.24.5 (#6234)
febb9d87c Union As Accessors for C# (#6251)
8778dc7c2 Resets buffer without deallocating current pointer (#6247)
aae376e9a Add GetBufferSpan() function to bufferbuilder (#6235)
0ff047148 Modernize android build and sample (#6229)
46a8c7e95 Added required-nested-flatbuffer to monster_test and fixed rust (#6236)
bc56c553e Notify based on Labelling issues and PR (#6241)
07d7cd78a Converted globs to use single quotes (#6240)
cdef70e24 More adjustments to the auto labeler (#6239)
9dd44df35 Updated Lua labeller glob (#6238)
c9b29d088 Support size-prefixed buffers and add tests for size-prefixed messages (#6232)
fba93e0ab Removes duplicate swift in labeler (#6228)
d1a545b1f Added more labels for auto labeler (#6227)
ea92a668d [C#] Optional Scalars (#6217)
6034de286 [Label Bot] Add Java and Kotlin support for the label bot (#6226)
b08b0a440 Implement `Debug` trait for Rust flatbuffers. (#6207)
17ae48dec [Label Bot] Adds some languages to labeler bot (#6222)
fc8097925 Auto Labeler Setup, Just C# for now (#6221)
55658f523 Auto Labeler Setup, Just C# for now (#6218)
14ecfe423 Updated comments and fixed a fundamental type error. (#6214)
a0182cdb1 optional scalars for ts/js (#6215)
0dfcc0a37 Adds NetTest.bat to run .NET Core tests on Windows (#6216)
f9a18ea63 [Java] Implement optional scalars (#6212)
c7586e85a Empties the sharedString map on reset on go and csharp (#6187)
914c64601 Removed C# references from java generator. Move annotations closer to definitions (#6204)
42d7c7997 Adds readable size to asserts in read functions (#6210)
e68e8d7de Refactor idl_gen_rust (#6206)
84809be7e Fix typo in flatbuffers::span declaration. (#6202)
1606fb637 Kotlin test optional enum (#6201)
fe8e3c7e5 Mass Refactoring to use `IsString` and other BASE_TYPE helpers (#6193)
8f6fa4b71 Updated SupportsAdvancedUnionFeatures to look out for string (#6190)
b46db38f5 [JS/TS] Rewrite flexbuffers JS to TS (#6148)
9fa1d2705 Rework enums in rust. (#6098)
a402b3aba idl_gen_json_schema Fix generation of arrays of enums (#6184)
0e1415b99 fix(go_test): remove deprecated grpc call (#6183)
5cd713710 Add generation of JSON Schema to library (#6165)
5be777e1d Bump junit from 4.12 to 4.13.1 in /grpc/tests (#6173)
a49d440ec Bump junit from 4.12 to 4.13.1 in /grpc (#6172)
4ec5e8db9 [C++] Add option to not generate direct copy methods. (#6166)
04bec23a3 Add Array initialization from struct constructor (#5865) (#6147)
77d57fd07 Cast to right type for reserved_ subtraction (#6167)
543c1bbeb Fixed rust nested flatbuffers for tables other than self (#6062)
cb971eece [C++] Fix -Wnarrowing and -Woverflow due to signed bitfields on G++ ARM (#6163)
7b9e61fcc [TS] GRPC Implementation (#6141)
3359e3042 Moved C++ to optional_scalars2 and added some tests. (#6162)
187a4787f [Rust] Upgrade flatbuffers library to 2018 edition (#6159)
08943aa26 Flatbuffer C++ UnpackTo optimization for vectors of non-bool bytes. (#6154)
5975658eb Enables optional enums in swift (#6160)
5d3cf440e Updated Lobster test for optional bools/enums
8ec8322f0 Ruopt enum (#6156)
bbcc85fd4 Fix generation of C++ code with Optional<Enum> (#6155)
0bdf2fa15 [C#] Fix and improve project files (#6142)
2eedc769d possibility to create a vector from an iterator (#6135)
ab01ae162 flatc should output a warning, when an attribute is attached more than once (#6146)
689bfafa7 [Python/JS/TS] Codegen SizeOf method for structs (#6136)
641309a5b unix2dos on tests/FlatBuffers.Test/FlatBuffers.Core.Test.csproj (#6133)
52e362879 SpanT is available in .Net Standard 2.0. (#6137)
dca12522a Add static cast to avoid implicit double promotion. (#6132)
e0bbaa6f9 [C#]Change to ENABLE_SPAN_T that doesn't require UNSAFE_BYTEBUFFER. (#6073)
ab139d6be Revert "[C#] Fix and improve project files (#6116)" (#6130)
34d67b425 Minireflect fixed array (#6129)
96d5e3597 [JS/TS] fix flatbuffers default export (#6123)
eb686a86f Add missed file generated by optional_scalar.fbs (#6125)
750281630 [C#] Fix and improve project files (#6116)
fb4e1c34f Add CharToLower and CharToUpper into util.s (#6126)
8c67b5b12 Add support of Optional<T> scalars to C++ code generator (#6092)
6228b66d3 [Kotlin] Support for optional scalars. (#6115)
e1be8aaad Bump version for latest swift version (#6121)
94873e595 [JS/TS] Modernize TypeScript / JavaScript flatbuffers support (#6095)
b8e87fafe [JS] Add getFullyQualifiedName() (#6119)
f96d1ef74 [Java] allowing larger buffer sizes when growing a byte buffer (#6118)
89435303b [Swift] Migrates struct write APIS to write directly to the buffer (#6093)
c75ae2429 Optional-ness in reflection (#6097)
338944d3d Rename Nullable scalars to Optional scalars (#6112)
f5ab24bc4 Avoid memcpy call for empty vectors (#6111)
92a8c1a0f [JS] FlexBuffers Fix for wrong type of offset and length values (#6107)
6cea45dcd fix c# json serializer commandline argument docs (#6104)
fec58aa12 Fix for issue 6100: incorrect shrinking logic in ResizeContext() (#6102)
71aca81ff [JS] FlexBuffers Support (#5973)
04d87ffec [C++] Small refactoring of the C++ code generator (#6091)
bb25956f0 Wrap verify file id condition in Check call (#6085)
49f4948f0 + Add `removable-media` plug to the snapcraft config (#6083)
eeacc53d2 Adds proper access types for swift object api & flatbuffers & grpc (#6081)
f3003e08d [Lobster] missed a test
d713a0084 [CMake] enabled multi-core builds in VS projects
77f966f89 [Lobster] optional scalars support
e86d5b8e9 [Kotlin] Attach JvmStatic annotation to each method in companion object (#6052)
db2aa9b4e [C#] Cleaned up .NET testing script for Mono (#6016)
63cc0eec4 Adds a serialize helper function to native table (#6059)
c30a87de6 [TS] Fix four bugs with imported types in TypeScript. (#6054)
a0fb30575 [Swift] Append namespace for Swift Grpc implementation (#6049)
77c18c1d6 export a __version__ variable for python module (#5309)
f1f23d08e adding fb import when no other imports are present (#6030)
f1025b284 [Feature] Checks for Nullable strings (#6050)
5d052f4e5 [Swift] RFC: Switch Swift namespace from public enum to ordinary concat with _ (#6045)
18b015d25 Rust codegen improvements and lint fixes (#6046)
d76e93f27 adds code gen for optional scalars in swift (#6038)
82fac326c [C++] Fix compiler error from deleted assignment operator (#6036) (#6047)
043b52bd4 Optional Scalars support for Rust (#6034)
c8fa0afdf Allow to run cpp tests under grpc/tests/ using bazel. (#6040)
6d0aae73c Fix git command executed in wrong folder when doing cmake superbuild (#6039)
ff1b73128 [Swift] Optional Scalars Preparation (#6028)
2e48c8dd3 tests: Check for both quiet and signaling NaN on mipsel/hppa (#6029)
6942704f2 support deprecated flag in json schema (#6022)
9ecd2e16c Flatc parser support for nullable scalars (#6026)
33e2d8079 [Dart] Generate constant values map for enums (#6025)
969d0f7a6 Using proper symbol name for reexport (#6021)
515a4052a Silence false positive "-Wstringop-overflow" on GCC 10.0 to 11.0 (#6020)
36fbe6f13 Updated FB import (#6019)
b69fc8cc9 [Java] Add support for shared strings on FlatBufferBuilder. (#6012)
ab6af18d9 Not using non-existent create method for obj api (#6015)
37a5dee10 Code cleanup + updates test and readme (#6004)
8a721f69a Serde with bytes maps to Blob (#6009)
e810635ea [Swift] FlatBuffers createMonster method doesn't treat struct properly (#5992)
4995e1527 Manage grpc dependency in Bazel's WORKSPACE file. (#5995)
60b6066fe Add warning to schema parser if field name is not snake_case. (#6005)
35d45cac7 [Rust] Flexbuffers dependency cleanup and fixes (#5998)
165a6e3d1 Re-added Evolution Schema Code Generation Command (#5999)
13d3fb2ea Fix RPM install conflict (#6003)
d64078eb2 [Swift] Initialize memory when clear ByteBuffer (#5982)
ca1190a3d [TS] Use proper TypedArray in create*Vector (#5991)
7571b2ac5 [C++] Updates real_path to be truly portable (#5787)
e5a8f76a4 [C++] Generate default member initializers for >= C++11 (#5989)
413bb9b55 [Kotlin] Fix Access to union of vector element (#5994)
f35184aef [Swift] Add parsing from unowned UnsafeMutableRawPointer for ByteBuffer (#5981)
b124b7625 Removed requirement that enums be declared in ascending order. (#5887)
0ec7600c6 Do not remove the last digit from float values (#5974)
14baf45c9 Mark GetBufferMinAlignment() const (#5985)
9abb2ec2c TypeScript/JavaScript docs improvements (#5984)
2e57d80b1 [Swift] Internal library improvements (#5965)
cfc7753a4 [Doc] Added missing letters to compiler options (#5976)
12ddc8a92 Rust Flexbuffers Documentation update (#5979)
24ad35709 [docs] typo: updates monsterdata.json to be valid json (#5978)
cc44a4442 [idl_parser] Mark typefield as deprecated (#5958)
9ab4a5c0e Deleted old stale bot
6682cfe87 Increased Operations per run in stale.yml
64922904b Adding Stale Action to clean up PR and Issues
8e505cb67 [C++] Fixed/Enabled --cpp-ptr-type std::shared_ptr [#5813] (#5959)
a28357d7a Propagate boolean default values from proto to fbs (#5964)
7cb4762a6 [Swift] Improving reallocation time by using memcpy and moving reallocation code to storage (#5960)
4e45f7c9e Fix error in SimpleQSort (#5955)
7ac026405 fix error on GRPC Python - ignore namespace tree if not specified (#5862) (#5922)
108e981db Required is now implemented in swift (#5952)
94a78e385 Fixed: Access violation and ASAN/UNSAN failures with sorted tables
53fb453e0 [rust] Add FlatBufferBuilder::force_defaults API (#5946)
17c1f35fa [FlexBuffer][Java] ReadWriteBuf and ReadBuf interface public (#5948)
2eaf57778 [Java] Grow ArrayReadWriteBuf enough to match requested capacity. (#5921)
666800da3 Adds bool support in structs + updates grpc support + CI upgrades (#5943)
38ed69eb3 fixed mutating inline values (#5942)
d026e6f07 Add static asserts to ensure that reflection API arrays are kept in sync (#5934)
988164f6e [C++] Got rid of memset's in constructors (#5938)
7179a5a8b General Codebase clean up (#5939)
a0da0c08c Add GetStringView like GetString, GetCstring (#5937)
ac203b209 [C#] Add file identifier to ObjectAPI Serialization Utility. (#5920)
8dd1bf25b not creating creation methods when using structs (#5919)
5aa443d98 [Dart] Adding FlexBuffers support (#5853)
0fa087657 [Dart] Getting tests/DartTest.sh to work on master. (#5915)
424a473e1 Schema parser: prohibit declaration of an array of pointers inside structs (#5907)
c3faa8346 Fix Cargo.toml dependencies (#5911)
91399ad05 fix union type names (#5902)
32782e4ad Update Rust Flexbuffers metadata before publishing (#5905)
e7f3b1690 [TS] Make Obj-API work with --short-names (#5898)
12ed1fe4a fixed invalid imports with --gen-all (#5895)
85ee4df7a [C#] Thread safe reads of Double and Float values from a ByteBuffer (#5900)
de89bd193 Implement flexbuffers in python (#5880)
8be05f6bd Rust Flexbuffers (#5669)
870ecbc09 [swift] Moves code to use VTablesStorage (#5888)
c2da8d5d8 [Java][FlexBuffers] Make FlexBuffersBuilder reusable by adding clear() (#5889) (#5890)
e84cbff67 Align package name to FindFlatBuffers.cmake (#5899)
f94e6c84e Small tutorial improvements - documentation only (#5894)
f12cca8bc Attempt at adding Github Actions CI
7e4124d6e Handle git program or .git folder absence (#5878)
a875d247a only add native_include_files if object based api is enabled (#5886)
6e9f5d981 Respect shared attribute in Parser (#5885)
ff1c78233 include/flatbuffers: typo fixes in comments (#5884)
2e9a19673 Updates swift docs for package management (#5883)
e3cb07d32 [Rust] idl_gen_rust.cpp: (Option/required-aware codegen for unions) (#5850)
712866d57 Propagate use_string_pooling in CopyTable (#5879)
44c919a9e Not using reexports with --gen-all (#5873)
99aa1ef21 Added INCLUDE_PREFIX option for flatbuffers_generate_headers (#5866)
40ba170c9 Fixed text in internals doc that implied structs can be root
cb4d0f72e [Swift] Object API support (#5826)
003e16405 [TS] Add Obj API (#5788)
21cf300f4 fix cpp usage markdown error (#5845)
9655e12d6 Upgraded swift implementation for grpc (#5843)
fb96fadc2 [C#] Fix nested structs and arrays in Object API (#5765)
408f11fbd [ts] Fix empty source/dest namespaces when reexporting. (#5841)
a83caf591 Improves performance for the swift library by using structs + a storage class (#5835)
925fab6b1 [Java][FlexBuffers] Optimize Map access (#5735)
d9fecc332 [CMake] : Add precompiled header support with FLATBUFFERS_ENABLE_PCH (#5827)
e9d453240 Added flatbuffers_generate_headers and flatbuffers_generate_binary_files cmake functions. (#5830)
c37c989ed Correct calculation of vector element size (#5831)
6b271b7ec Fix Clang-trunk warnings about special members deprecated in C++20. (#5829)
90f3b8e8c Fix `float_constant` definition in './doc/Grammar.md` (#5828)
3af735934 [csharp] flatc should generate a 'Create…' method for tables with struct field… (#5818)
c4231c3cb Updated doxyfile - added missing files (#5824)
9657df184 Update Grammar.md (#5820)
97ffc590e Include CPack only when explictly packaging (#5817)
8b52af65b [C++] Add max_depth and max_tables parameters to reflection::Verify (#5815)
9b034eee1 Fix interpretation of 'nan(number)' by the idl_parser (#5810)
3e9ac3cff [Scripts] Adds swift to generated code (#5806)
697147a2e updated maven build files
6df40a247 pre-tag version bump for 1.12
0dba63909 Removes the inner loop in the endtable check written tables (#5803)
0e3fdd0ee Escape characters in jsonschema descriptions (#5644)
45a2b07cb Remove `noexcept` qualifier copy-ctor of `union` type (#5800) (#5802)
d10c16314 Replace 'assert' by 'FLATBUFFERS_ASSERT' inside idl_parser.cpp (#5799)
35abb7f89 Add non-nullable modifier to return type of functions never returning null (#5797)
9954e09ab [C++] Generate code for vector force_align attribute. (#5796)
95a21327f rust: pub export the VectorIter type (#5736)
89b6183ee Fix Python min alignment
5a98d65e8 [Rust] Add gen-name-strings for Rust (#5757)
f73d205bc Removed assert that wasn't backwards compatible.
7c37abe92 [C#] add ObjectAPI Serialization Utility (#5785)
4749e77b0 Fixed docs on how to obtain parser error.
6ff189841 Added --filename-suffix and --filename-ext to flatc (#5778)
c9a30c9ca Fixed refractoring issue in reflection/generate_code.sh. Also, mv deletes the original file, so I don't need to clean it up manually in that case. (#5777)
8c02d17be Skip writing reflection_generated.h if not changed (#5776)
34305c4ce [Swift] Adds GRPC to Swift (#5758)
cd88e6b2a [Java][FlexBuffers] Abstract buffer access from ByteBuffer (#5743)
3ec7a53c6 Adds cocoapods and a readme of how to get the package (#5771)
6d44cede7 [snap] Fix versioning (#5727)
cc08c0835 [Python] Fixed potential allignment issue (#5768)
54f8b787c Fix memory leak on cpp object api (#5761)
17557f913 [Python] Fixed issue #5499 (#5764)
d54af8cd4 [C++] Use strong enum type for vectors when scoped-enums is on. (#5750)
173e10fdf [C#] support Json Serialization (#5752)
8f56990f6 FlexBuffers: C++: scalar-only typed vectors were not aligned.
6400c9b05 Bump Rust port to 0.6.1 (#5747)
7418d8587 [C#] support Object API (#5710)
c580fa284 Adds min and max, comments, and all of swift's keywords + fix docs (#5737)
f2a127230 Use VS 2017 and 2019 on CI, fix cast issue in dart_idl (#5740)
316d7c208 Creates a flatbuffers validation function + small fix (#5725)
47026ea6b Added the code to embed the binary schema to the source (--bfbs-gen-embed). (#5701)
3f677f241 [Java][FlexBuffers] Deprecate typed vector strings due to design flaw (#5722)
a593a11e5 [Go] Implements a SharedStrings function (#5733)
7cdfc8475 [Swift] Fix padding function overflow when bufSize is 0 (#5721)
bab2b0db4 Add vcpkg installation instructions (#5732)
89418eb84 [Dart] Fix deprecated field support, inf/nan (#5724)
9cadf05d8 [typescript] Size-prefixed root accessors not taking into account size prefix (#5717)
6da1cf79d [rust] Add use declarations to Rust-generated bindings for imported FB definitions (#5645)
bee1df96d [Go] Replace references to hardcoded ”Monster" etc with idiomatic go wherever possible (#5716)
01189d7ed [C++] Fix for printing of enum in case output_enum_identifiers=1. (#5706)
c4b2b0a25 [Swift] Support create long string (#5709)
a4b2884e4 Added create function for swift (#5707)
04d80f255 [Swift] Swift implementation 🎉🎉 (#5603)
55686100a Changed direct calls to strtod to use StringToNumber
718351831 Document JSON compatibility guarantees. (#5704)
d1b34f0f2 Add CMake 'generated_code' target to simplify resolution of build dependencies (#5697)
21b706196 (Optionally) add an additional suffix namespace to generated fbs files. (#5698)
35daaf83d [Java] Replace Table.UTF8_CHARSET with StandardCharsets.UTF_8 (#5696)
3b458f7a1 Rust: Temporarily disable 2 endianness unit tests (#5695)
a5d9d0f7d [C++17] Add Traits class for Tables and Factory function within it. (#5678)
3cd9b6434 Removed code_generators.cpp from library targets
355dfd48d [rust] Make enum names public (#5690)
bcd58a159 Correct inverted logic around include prefixes. (#5689)
a2c12900a Optimize Pack method using numpy (#5662)
901b89e73 [C++] Add Builder and Table typedefs (#5685)
31f879908 Minor doc updates: FlexBuffers C#, Discord, CppUsage.
8023d99e2 Upgrade rules_go (#5684)
b4154405d Fix --incompatible_load_cc_rules_from_bzl (#5683)
04c17c7a7 Add support for absl::string_view when available (#5682)
62ec7d52c [Bazel] Add support for compatible_with and restricted_to (#5681)
7de668053 CI: New Docker tests for Python with numpy (#5677)
3a70e0b30 Fixed struct initialization error on older versions of C#
9b1320135 Fixed warnings in FlexBuffers.java
5e3916050 Fixed out of date licenses on gRPC Python files.
c95755051 Removed test proto output.
44bf71988 Add flatc '--cpp_std' switch (#5656)
3e8f15df9 Fix for FlexBuffers FBT_VECTOR_STRING size bit-width.
602721a73 Added Check to VerifyAlignment (#5675)
13c05f4da Improve import handling for proto conversion (#5673)
ce3a1c43a [Dart] Fix prepare space for writeListInt64 and writeListUint64 (#5654)
aa75e5734 Make Rust constants public (#5659)
2790fee25 Add namespace qualification to union types (#5666)
eddebec1b Bugfix for Rust generation of union fields named with language keywords (#5592)
030fee36a wrap multiple statements in do {} while(!IsConstTrue(true)) (#5655)
f9724d1bd [gRPC] Uncomment MessageBuilder (#5658)
b20801ca4 Supress unsigned-integer-overflow for PaddingBytes (#5647)
a8e800bd7 Add --force-empty-vectors option (#5653)
d7530ae96 Fixed enum min/max values not properly escaped.
99d11e279 Split Bazel targets into multiple packages (#5640)
4fd8eb214 Remove a static_assert (#5643)
65f870357 Flatbuffers Python Object API (#5616)
75823cc27 [Clang 10]: definition of implicit copy constructor for 'TableKeyComparatoris deprecated #5649 (#5650)
58e279244 [docs]: add missing semicolon (#5648)
3c964e10a [GO] Fix support for enums with underscores and Unions with imported members (#5600)
c3c32ec94 Fix ambiguity of a type deduction in TEST_EQ macro if arguments have `enum class` type. (#5630)
075e8d676 Simplify declarations of x-macro FLATBUFFERS_TD (#5638)
bcf1bd5c9 read vtable size through ReadScalar() (#5636)
136d75fa6 Changed null checks in test. Removed verifier pointer usage (#5634)
091fa1fd1 Add testing of C++ with sanitizers (CI-Docker) (#5631)
ff3781dc2 add namespace prefix in FLATBUFFERS_MAX_BUFFER_SIZE (#5629)
6beb9f49c Support for python grpc - continuing the work from the pull request #4270 #4705 (#5613)
80988ea86 Removed idl_gen_general.cpp and move contents to code_generators.cpp (#5625)
0f2ff7eaa Lua cleanup (#5624)
dda095023 [C++] Adds basic schema evolution tests (#5611)
adbcbba5d [C++, C#, Java] Separated C# and Java generators into their own classes (#5618)
cbbd6aca0 add check for root_type specified for json schema generation (#5622)
405c64e07 [Rust] Bump smallvec version to 1.0 (#5621)
42c08cbca Ran src/clang-format-all.sh (#5617)
33d5dd9bd Improved pull request & clang-format instructions.
105dd528e Change monster_extra generation to use flatbuffers::unique_ptr (#5612)
f0f0efe7b [C++] Refactor to conform to Google C++ style guide (#5608)
e837d5a29 Fixed deprecated method in GRPC Java test.
9834ee978 Fixed Apache license not using canonical version.
44b2ab087 include/flatbuffers/base.h: fix no_sanitize issue with old clang (#5610)
46ae3f80a [C++, Java, C#, TypeScript, JavaScript] Skip generation of mutable union types (#5599)
7b38aa71e flatbuffers.h: fix documentation warning (#5607)
661bedd83 Add Lua FlatbufferBuilder Clean() method to enable reuseable builders (#5606)
8526e12d7 [Kotlin] Fix union vector accessor after change in Java API (#5605)
3c7b660d6 [flatc] Remove an always true condition for flexbuffers (#5604)
964365ba6 [Go] Add UnPackTo functions (#5598)
32254b7ac [Go] Object API support (#5339)
521e255ad Rust: Add idiomatic iterator for Vector type (#5579)
1b85292fd Fix typos in comments (#5590)
480815447 C++ verifier for evolved union fields should return true (#5586)
8d5e424c6 Add ByteBuffer copy for vector of bytes in Java (#5587)
b4774d235 Rust: Fix Copy and Clone impls for a few generic types (#5577)
26f238c24 Add `--clean-first` to the cmake-build command (travis) (#5574)
e93c8c46e Fix Follow implementation for bool (#5554)
e21516b9d Fix issue #5557 (#5573)
fbc11e8ae Avoid intentional unsigned integer overflow getting caught by sanitizers (#5572)
e9d29c21a Python: Add forceDefaults opt to python Builder (#5564)
8bfafc76d Java: Don't annotate vector-of-tables item getters with @nullable. (#5562)
df3e8bf4a Fixed warnings generated by recent JSON sorting feature.
5665cfe49 [Java] byte buffer factory returned buffer capcity is used instead of the requested size (#5558)
5797540ed #5544 Fix of Array of table is not sorted if Create<type>Direct() is used (#5546)
7f1af7cb0 Fix build with gcc version 7.4.0 (#5570)
32f47ad24 Fixed JSON parser not sorting vectors of tables/structs with key.
842f672ba [FlexBuffers][Java] Cache size of Sized objects in FlexBuffers (#5551)
d4cae0a62 Fix issue #5542 (#5543)
f1147f65b Fixed Android STLPort related error.
69d3fec48 Fix namespaced struct/field name collision detection (#5540) (#5545)
cfb4ecf6f [flac] Add FlexBuffers option for generating data (#5519)
a92039687 Update Rust versions under test from 1.30.1 to 1.37.0 (#5538)
625338d09 Adds XOPEN_SOURCE for PATH_MAX and POSIX 1993 for stat (#5529)
3f8ce99c5 [FlexBuffers][Java] Add override Key::toString (#5533)
0798b7b69 [FlexBuffers][Java] Fix wrong access to a string using Reference::asString(). (#5532)
cbdf82e2f Fix Mutate() methods of Array<scalar/struct> (override 5508) (#5526)
e365c502f Java: Added access object for vector of struct and vector of tables. (#5233)
97f3aa917 Fixed DetachedBuffer self move assignment (#5521)
2f5bb2eec Fix buildifier warnings found in new bazel (#5517)
917687c7a Fixed Reflection Verifier not handling vectors of unions.
f9277e691 Fixed GenerateText not handling vectors of unions.
2706381ee Add element size parameter to __vector_as_arraysegment [c#] (#5512)
b5560fcd5 [Java][FlexBuffers] Improve documentation for FlexBuffers in Java. (#5506)
782b865c5 Annotate getters with @Pure when --java-checkerframework is specified. (#5510)
3bfc86eaf [Dart]fix: segment fault with empty namespace when generating dart file (#5507)
c0282873f Rust: Fixed cargo clippy on non-generated code (#5485)
4b870aca9 [Javascript] Fix syntax error for signed enum (#5503)
d0e3870c0 [C#] Fix retrieving enumeration vectors as arrays (#5457)
fb25eb87f Doc typo fixes (#5505)
cb35d3a0e Use all of the available space in the buffer returned by ByteBufferFactory to allow the factory to keep a pool of larger than initialsize sized buffers. (#5500)
8e6cabb31 [FlexBuffers][Java] Implementation of FlexBuffers API (#5476)
bd31dd242 Clarified value reuse in FlexBuffers
65b67d213 Fixed test build invocation of arrays_test.fbs
1fbb71132 FlexBuffers: allow any values to be shared.
cd75a3658 Android: remove app_dummy() calls
ec6b0bf29 Fixed STLPort Android compile error
c11b5d744 [bugfix]flexbuffers isvector bugfix (#5488)
4525c91be Fix incorrect padding in arrays of structs (Issue #5484) (#5491)
b97b342f5 Fixed missing generated code.
c1058a903 C++ IDL generation adds superfluous semicolon in GenTablePost, causing (#5483)
303044934 [go]add Name() for ForceCodec interface (#5486)
a2485d4ec reflection: check for valid union enum value during object verification (#5475)
a20e71ac9 has_method support for primitive fields in java runtime. Changed: idl.h, FlatBufferBuilder.java , idl_gen_general.cpp, idl_parser.cpp, flatc.cpp (#5468)
Change-Id: I836f4b43e6818bb16425a87899e6234ac86242aa
git-subtree-dir: third_party/flatbuffers
git-subtree-split: d6a8dbd26ff08a8868e0d0c1b4b67d31b40e4a7f
diff --git a/python/flatbuffers/__init__.py b/python/flatbuffers/__init__.py
index d14872a..74dc7ee 100644
--- a/python/flatbuffers/__init__.py
+++ b/python/flatbuffers/__init__.py
@@ -15,3 +15,5 @@
from .builder import Builder
from .table import Table
from .compat import range_func as compat_range
+from ._version import __version__
+from . import util
diff --git a/python/flatbuffers/_version.py b/python/flatbuffers/_version.py
new file mode 100644
index 0000000..a44e900
--- /dev/null
+++ b/python/flatbuffers/_version.py
@@ -0,0 +1,17 @@
+# Copyright 2019 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Placeholder, to be updated during the release process
+# by the setup.py
+__version__ = u"latest"
diff --git a/python/flatbuffers/builder.py b/python/flatbuffers/builder.py
index d04ee85..f9448b0 100644
--- a/python/flatbuffers/builder.py
+++ b/python/flatbuffers/builder.py
@@ -103,7 +103,7 @@
## @cond FLATBUFFERS_INTENRAL
__slots__ = ("Bytes", "current_vtable", "head", "minalign", "objectEnd",
- "vtables", "nested", "finished")
+ "vtables", "nested", "forceDefaults", "finished")
"""Maximum buffer size constant, in bytes.
@@ -131,10 +131,10 @@
self.objectEnd = None
self.vtables = {}
self.nested = False
+ self.forceDefaults = False
## @endcond
self.finished = False
-
def Output(self):
"""Return the portion of the buffer that has been used for writing data.
@@ -466,7 +466,7 @@
# tobytes ensures c_contiguous ordering
self.Bytes[self.Head():self.Head()+l] = x_lend.tobytes(order='C')
-
+
return self.EndVector(x.size)
## @cond FLATBUFFERS_INTERNAL
@@ -514,17 +514,24 @@
"""Finish finalizes a buffer, pointing to the given `rootTable`."""
N.enforce_number(rootTable, N.UOffsetTFlags)
+ prepSize = N.UOffsetTFlags.bytewidth
if file_identifier is not None:
- self.Prep(N.UOffsetTFlags.bytewidth, N.Uint8Flags.bytewidth*4)
-
+ prepSize += N.Int32Flags.bytewidth
+ if sizePrefix:
+ prepSize += N.Int32Flags.bytewidth
+ self.Prep(self.minalign, prepSize)
+
+ if file_identifier is not None:
+ self.Prep(N.UOffsetTFlags.bytewidth, encode.FILE_IDENTIFIER_LENGTH)
+
# Convert bytes object file_identifier to an array of 4 8-bit integers,
# and use big-endian to enforce size compliance.
# https://docs.python.org/2/library/struct.html#format-characters
file_identifier = N.struct.unpack(">BBBB", file_identifier)
for i in range(encode.FILE_IDENTIFIER_LENGTH-1, -1, -1):
# Place the bytes of the file_identifer in reverse order:
- self.Place(file_identifier[i], N.Uint8Flags)
-
+ self.Place(file_identifier[i], N.Uint8Flags)
+
self.PrependUOffsetTRelative(rootTable)
if sizePrefix:
size = len(self.Bytes) - self.Head()
@@ -552,7 +559,7 @@
def PrependSlot(self, flags, o, x, d):
N.enforce_number(x, flags)
N.enforce_number(d, flags)
- if x != d:
+ if x != d or self.forceDefaults:
self.Prepend(flags, x)
self.Slot(o)
@@ -589,7 +596,7 @@
be set to zero and no other data will be written.
"""
- if x != d:
+ if x != d or self.forceDefaults:
self.PrependUOffsetTRelative(x)
self.Slot(o)
@@ -691,6 +698,15 @@
"""
self.Prepend(N.Float64Flags, x)
+ def ForceDefaults(self, forceDefaults):
+ """
+ In order to save space, fields that are set to their default value
+ don't get serialized into the buffer. Forcing defaults provides a
+ way to manually disable this optimization. When set to `True`, will
+ always serialize default values.
+ """
+ self.forceDefaults = forceDefaults
+
##############################################################
## @cond FLATBUFFERS_INTERNAL
diff --git a/python/flatbuffers/flexbuffers.py b/python/flatbuffers/flexbuffers.py
new file mode 100644
index 0000000..da10668
--- /dev/null
+++ b/python/flatbuffers/flexbuffers.py
@@ -0,0 +1,1527 @@
+# Lint as: python3
+# Copyright 2020 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Implementation of FlexBuffers binary format.
+
+For more info check https://google.github.io/flatbuffers/flexbuffers.html and
+corresponding C++ implementation at
+https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flexbuffers.h
+"""
+
+# pylint: disable=invalid-name
+# TODO(dkovalev): Add type hints everywhere, so tools like pytypes could work.
+
+import array
+import contextlib
+import enum
+import struct
+
+__all__ = ('Type', 'Builder', 'GetRoot', 'Dumps', 'Loads')
+
+
+class BitWidth(enum.IntEnum):
+ """Supported bit widths of value types.
+
+ These are used in the lower 2 bits of a type field to determine the size of
+ the elements (and or size field) of the item pointed to (e.g. vector).
+ """
+ W8 = 0 # 2^0 = 1 byte
+ W16 = 1 # 2^1 = 2 bytes
+ W32 = 2 # 2^2 = 4 bytes
+ W64 = 3 # 2^3 = 8 bytes
+
+ @staticmethod
+ def U(value):
+ """Returns the minimum `BitWidth` to encode unsigned integer value."""
+ assert value >= 0
+
+ if value < (1 << 8):
+ return BitWidth.W8
+ elif value < (1 << 16):
+ return BitWidth.W16
+ elif value < (1 << 32):
+ return BitWidth.W32
+ elif value < (1 << 64):
+ return BitWidth.W64
+ else:
+ raise ValueError('value is too big to encode: %s' % value)
+
+ @staticmethod
+ def I(value):
+ """Returns the minimum `BitWidth` to encode signed integer value."""
+ # -2^(n-1) <= value < 2^(n-1)
+ # -2^n <= 2 * value < 2^n
+ # 2 * value < 2^n, when value >= 0 or 2 * (-value) <= 2^n, when value < 0
+ # 2 * value < 2^n, when value >= 0 or 2 * (-value) - 1 < 2^n, when value < 0
+ #
+ # if value >= 0:
+ # return BitWidth.U(2 * value)
+ # else:
+ # return BitWidth.U(2 * (-value) - 1) # ~x = -x - 1
+ value *= 2
+ return BitWidth.U(value if value >= 0 else ~value)
+
+ @staticmethod
+ def F(value):
+ """Returns the `BitWidth` to encode floating point value."""
+ if struct.unpack('f', struct.pack('f', value))[0] == value:
+ return BitWidth.W32
+ return BitWidth.W64
+
+ @staticmethod
+ def B(byte_width):
+ return {
+ 1: BitWidth.W8,
+ 2: BitWidth.W16,
+ 4: BitWidth.W32,
+ 8: BitWidth.W64
+ }[byte_width]
+
+
+I = {1: 'b', 2: 'h', 4: 'i', 8: 'q'} # Integer formats
+U = {1: 'B', 2: 'H', 4: 'I', 8: 'Q'} # Unsigned integer formats
+F = {4: 'f', 8: 'd'} # Floating point formats
+
+
+def _Unpack(fmt, buf):
+ return struct.unpack(fmt[len(buf)], buf)[0]
+
+
+def _UnpackVector(fmt, buf, length):
+ byte_width = len(buf) // length
+ return struct.unpack('%d%s' % (length, fmt[byte_width]), buf)
+
+
+def _Pack(fmt, value, byte_width):
+ return struct.pack(fmt[byte_width], value)
+
+
+def _PackVector(fmt, values, byte_width):
+ return struct.pack('%d%s' % (len(values), fmt[byte_width]), *values)
+
+
+def _Mutate(fmt, buf, value, byte_width, value_bit_width):
+ if (1 << value_bit_width) <= byte_width:
+ buf[:byte_width] = _Pack(fmt, value, byte_width)
+ return True
+ return False
+
+
+# Computes how many bytes you'd have to pad to be able to write an
+# "scalar_size" scalar if the buffer had grown to "buf_size",
+# "scalar_size" is a power of two.
+def _PaddingBytes(buf_size, scalar_size):
+ # ((buf_size + (scalar_size - 1)) // scalar_size) * scalar_size - buf_size
+ return -buf_size & (scalar_size - 1)
+
+
+def _ShiftSlice(s, offset, length):
+ start = offset + (0 if s.start is None else s.start)
+ stop = offset + (length if s.stop is None else s.stop)
+ return slice(start, stop, s.step)
+
+
+# https://en.cppreference.com/w/cpp/algorithm/lower_bound
+def _LowerBound(values, value, pred):
+ """Implementation of C++ std::lower_bound() algorithm."""
+ first, last = 0, len(values)
+ count = last - first
+ while count > 0:
+ i = first
+ step = count // 2
+ i += step
+ if pred(values[i], value):
+ i += 1
+ first = i
+ count -= step + 1
+ else:
+ count = step
+ return first
+
+
+# https://en.cppreference.com/w/cpp/algorithm/binary_search
+def _BinarySearch(values, value, pred=lambda x, y: x < y):
+ """Implementation of C++ std::binary_search() algorithm."""
+ index = _LowerBound(values, value, pred)
+ if index != len(values) and not pred(value, values[index]):
+ return index
+ return -1
+
+
+class Type(enum.IntEnum):
+ """Supported types of encoded data.
+
+ These are used as the upper 6 bits of a type field to indicate the actual
+ type.
+ """
+ NULL = 0
+ INT = 1
+ UINT = 2
+ FLOAT = 3
+ # Types above stored inline, types below store an offset.
+ KEY = 4
+ STRING = 5
+ INDIRECT_INT = 6
+ INDIRECT_UINT = 7
+ INDIRECT_FLOAT = 8
+ MAP = 9
+ VECTOR = 10 # Untyped.
+
+ VECTOR_INT = 11 # Typed any size (stores no type table).
+ VECTOR_UINT = 12
+ VECTOR_FLOAT = 13
+ VECTOR_KEY = 14
+ # DEPRECATED, use VECTOR or VECTOR_KEY instead.
+ # Read test.cpp/FlexBuffersDeprecatedTest() for details on why.
+ VECTOR_STRING_DEPRECATED = 15
+
+ VECTOR_INT2 = 16 # Typed tuple (no type table, no size field).
+ VECTOR_UINT2 = 17
+ VECTOR_FLOAT2 = 18
+ VECTOR_INT3 = 19 # Typed triple (no type table, no size field).
+ VECTOR_UINT3 = 20
+ VECTOR_FLOAT3 = 21
+ VECTOR_INT4 = 22 # Typed quad (no type table, no size field).
+ VECTOR_UINT4 = 23
+ VECTOR_FLOAT4 = 24
+
+ BLOB = 25
+ BOOL = 26
+ VECTOR_BOOL = 36 # To do the same type of conversion of type to vector type
+
+ @staticmethod
+ def Pack(type_, bit_width):
+ return (int(type_) << 2) | bit_width
+
+ @staticmethod
+ def Unpack(packed_type):
+ return 1 << (packed_type & 0b11), Type(packed_type >> 2)
+
+ @staticmethod
+ def IsInline(type_):
+ return type_ <= Type.FLOAT or type_ == Type.BOOL
+
+ @staticmethod
+ def IsTypedVector(type_):
+ return Type.VECTOR_INT <= type_ <= Type.VECTOR_STRING_DEPRECATED or \
+ type_ == Type.VECTOR_BOOL
+
+ @staticmethod
+ def IsTypedVectorElementType(type_):
+ return Type.INT <= type_ <= Type.STRING or type_ == Type.BOOL
+
+ @staticmethod
+ def ToTypedVectorElementType(type_):
+ if not Type.IsTypedVector(type_):
+ raise ValueError('must be typed vector type')
+
+ return Type(type_ - Type.VECTOR_INT + Type.INT)
+
+ @staticmethod
+ def IsFixedTypedVector(type_):
+ return Type.VECTOR_INT2 <= type_ <= Type.VECTOR_FLOAT4
+
+ @staticmethod
+ def IsFixedTypedVectorElementType(type_):
+ return Type.INT <= type_ <= Type.FLOAT
+
+ @staticmethod
+ def ToFixedTypedVectorElementType(type_):
+ if not Type.IsFixedTypedVector(type_):
+ raise ValueError('must be fixed typed vector type')
+
+ # 3 types each, starting from length 2.
+ fixed_type = type_ - Type.VECTOR_INT2
+ return Type(fixed_type % 3 + Type.INT), fixed_type // 3 + 2
+
+ @staticmethod
+ def ToTypedVector(element_type, fixed_len=0):
+ """Converts element type to corresponding vector type.
+
+ Args:
+ element_type: vector element type
+ fixed_len: number of elements: 0 for typed vector; 2, 3, or 4 for fixed
+ typed vector.
+
+ Returns:
+ Typed vector type or fixed typed vector type.
+ """
+ if fixed_len == 0:
+ if not Type.IsTypedVectorElementType(element_type):
+ raise ValueError('must be typed vector element type')
+ else:
+ if not Type.IsFixedTypedVectorElementType(element_type):
+ raise ValueError('must be fixed typed vector element type')
+
+ offset = element_type - Type.INT
+ if fixed_len == 0:
+ return Type(offset + Type.VECTOR_INT) # TypedVector
+ elif fixed_len == 2:
+ return Type(offset + Type.VECTOR_INT2) # FixedTypedVector
+ elif fixed_len == 3:
+ return Type(offset + Type.VECTOR_INT3) # FixedTypedVector
+ elif fixed_len == 4:
+ return Type(offset + Type.VECTOR_INT4) # FixedTypedVector
+ else:
+ raise ValueError('unsupported fixed_len: %s' % fixed_len)
+
+
+class Buf:
+ """Class to access underlying buffer object starting from the given offset."""
+
+ def __init__(self, buf, offset):
+ self._buf = buf
+ self._offset = offset if offset >= 0 else len(buf) + offset
+ self._length = len(buf) - self._offset
+
+ def __getitem__(self, key):
+ if isinstance(key, slice):
+ return self._buf[_ShiftSlice(key, self._offset, self._length)]
+ elif isinstance(key, int):
+ return self._buf[self._offset + key]
+ else:
+ raise TypeError('invalid key type')
+
+ def __setitem__(self, key, value):
+ if isinstance(key, slice):
+ self._buf[_ShiftSlice(key, self._offset, self._length)] = value
+ elif isinstance(key, int):
+ self._buf[self._offset + key] = key
+ else:
+ raise TypeError('invalid key type')
+
+ def __repr__(self):
+ return 'buf[%d:]' % self._offset
+
+ def Find(self, sub):
+ """Returns the lowest index where the sub subsequence is found."""
+ return self._buf[self._offset:].find(sub)
+
+ def Slice(self, offset):
+ """Returns new `Buf` which starts from the given offset."""
+ return Buf(self._buf, self._offset + offset)
+
+ def Indirect(self, offset, byte_width):
+ """Return new `Buf` based on the encoded offset (indirect encoding)."""
+ return self.Slice(offset - _Unpack(U, self[offset:offset + byte_width]))
+
+
+class Object:
+ """Base class for all non-trivial data accessors."""
+ __slots__ = '_buf', '_byte_width'
+
+ def __init__(self, buf, byte_width):
+ self._buf = buf
+ self._byte_width = byte_width
+
+ @property
+ def ByteWidth(self):
+ return self._byte_width
+
+
+class Sized(Object):
+ """Base class for all data accessors which need to read encoded size."""
+ __slots__ = '_size',
+
+ def __init__(self, buf, byte_width, size=0):
+ super().__init__(buf, byte_width)
+ if size == 0:
+ self._size = _Unpack(U, self.SizeBytes)
+ else:
+ self._size = size
+
+ @property
+ def SizeBytes(self):
+ return self._buf[-self._byte_width:0]
+
+ def __len__(self):
+ return self._size
+
+
+class Blob(Sized):
+ """Data accessor for the encoded blob bytes."""
+ __slots__ = ()
+
+ @property
+ def Bytes(self):
+ return self._buf[0:len(self)]
+
+ def __repr__(self):
+ return 'Blob(%s, size=%d)' % (self._buf, len(self))
+
+
+class String(Sized):
+ """Data accessor for the encoded string bytes."""
+ __slots__ = ()
+
+ @property
+ def Bytes(self):
+ return self._buf[0:len(self)]
+
+ def Mutate(self, value):
+ """Mutates underlying string bytes in place.
+
+ Args:
+ value: New string to replace the existing one. New string must have less
+ or equal UTF-8-encoded bytes than the existing one to successfully
+ mutate underlying byte buffer.
+
+ Returns:
+ Whether the value was mutated or not.
+ """
+ encoded = value.encode('utf-8')
+ n = len(encoded)
+ if n <= len(self):
+ self._buf[-self._byte_width:0] = _Pack(U, n, self._byte_width)
+ self._buf[0:n] = encoded
+ self._buf[n:len(self)] = bytearray(len(self) - n)
+ return True
+ return False
+
+ def __str__(self):
+ return self.Bytes.decode('utf-8')
+
+ def __repr__(self):
+ return 'String(%s, size=%d)' % (self._buf, len(self))
+
+
+class Key(Object):
+ """Data accessor for the encoded key bytes."""
+ __slots__ = ()
+
+ def __init__(self, buf, byte_width):
+ assert byte_width == 1
+ super().__init__(buf, byte_width)
+
+ @property
+ def Bytes(self):
+ return self._buf[0:len(self)]
+
+ def __len__(self):
+ return self._buf.Find(0)
+
+ def __str__(self):
+ return self.Bytes.decode('ascii')
+
+ def __repr__(self):
+ return 'Key(%s, size=%d)' % (self._buf, len(self))
+
+
+class Vector(Sized):
+ """Data accessor for the encoded vector bytes."""
+ __slots__ = ()
+
+ def __getitem__(self, index):
+ if index < 0 or index >= len(self):
+ raise IndexError('vector index %s is out of [0, %d) range' % \
+ (index, len(self)))
+
+ packed_type = self._buf[len(self) * self._byte_width + index]
+ buf = self._buf.Slice(index * self._byte_width)
+ return Ref.PackedType(buf, self._byte_width, packed_type)
+
+ @property
+ def Value(self):
+ """Returns the underlying encoded data as a list object."""
+ return [e.Value for e in self]
+
+ def __repr__(self):
+ return 'Vector(%s, byte_width=%d, size=%d)' % \
+ (self._buf, self._byte_width, self._size)
+
+
+class TypedVector(Sized):
+ """Data accessor for the encoded typed vector or fixed typed vector bytes."""
+ __slots__ = '_element_type', '_size'
+
+ def __init__(self, buf, byte_width, element_type, size=0):
+ super().__init__(buf, byte_width, size)
+
+ if element_type == Type.STRING:
+ # These can't be accessed as strings, since we don't know the bit-width
+ # of the size field, see the declaration of
+ # FBT_VECTOR_STRING_DEPRECATED above for details.
+ # We change the type here to be keys, which are a subtype of strings,
+ # and will ignore the size field. This will truncate strings with
+ # embedded nulls.
+ element_type = Type.KEY
+
+ self._element_type = element_type
+
+ @property
+ def Bytes(self):
+ return self._buf[:self._byte_width * len(self)]
+
+ @property
+ def ElementType(self):
+ return self._element_type
+
+ def __getitem__(self, index):
+ if index < 0 or index >= len(self):
+ raise IndexError('vector index %s is out of [0, %d) range' % \
+ (index, len(self)))
+
+ buf = self._buf.Slice(index * self._byte_width)
+ return Ref(buf, self._byte_width, 1, self._element_type)
+
+ @property
+ def Value(self):
+ """Returns underlying data as list object."""
+ if not self:
+ return []
+
+ if self._element_type is Type.BOOL:
+ return [bool(e) for e in _UnpackVector(U, self.Bytes, len(self))]
+ elif self._element_type is Type.INT:
+ return list(_UnpackVector(I, self.Bytes, len(self)))
+ elif self._element_type is Type.UINT:
+ return list(_UnpackVector(U, self.Bytes, len(self)))
+ elif self._element_type is Type.FLOAT:
+ return list(_UnpackVector(F, self.Bytes, len(self)))
+ elif self._element_type is Type.KEY:
+ return [e.AsKey for e in self]
+ elif self._element_type is Type.STRING:
+ return [e.AsString for e in self]
+ else:
+ raise TypeError('unsupported element_type: %s' % self._element_type)
+
+ def __repr__(self):
+ return 'TypedVector(%s, byte_width=%d, element_type=%s, size=%d)' % \
+ (self._buf, self._byte_width, self._element_type, self._size)
+
+
+class Map(Vector):
+ """Data accessor for the encoded map bytes."""
+
+ @staticmethod
+ def CompareKeys(a, b):
+ if isinstance(a, Ref):
+ a = a.AsKeyBytes
+ if isinstance(b, Ref):
+ b = b.AsKeyBytes
+ return a < b
+
+ def __getitem__(self, key):
+ if isinstance(key, int):
+ return super().__getitem__(key)
+
+ index = _BinarySearch(self.Keys, key.encode('ascii'), self.CompareKeys)
+ if index != -1:
+ return super().__getitem__(index)
+
+ raise KeyError(key)
+
+ @property
+ def Keys(self):
+ byte_width = _Unpack(U, self._buf[-2 * self._byte_width:-self._byte_width])
+ buf = self._buf.Indirect(-3 * self._byte_width, self._byte_width)
+ return TypedVector(buf, byte_width, Type.KEY)
+
+ @property
+ def Values(self):
+ return Vector(self._buf, self._byte_width)
+
+ @property
+ def Value(self):
+ return {k.Value: v.Value for k, v in zip(self.Keys, self.Values)}
+
+ def __repr__(self):
+ return 'Map(%s, size=%d)' % (self._buf, len(self))
+
+
+class Ref:
+ """Data accessor for the encoded data bytes."""
+ __slots__ = '_buf', '_parent_width', '_byte_width', '_type'
+
+ @staticmethod
+ def PackedType(buf, parent_width, packed_type):
+ byte_width, type_ = Type.Unpack(packed_type)
+ return Ref(buf, parent_width, byte_width, type_)
+
+ def __init__(self, buf, parent_width, byte_width, type_):
+ self._buf = buf
+ self._parent_width = parent_width
+ self._byte_width = byte_width
+ self._type = type_
+
+ def __repr__(self):
+ return 'Ref(%s, parent_width=%d, byte_width=%d, type_=%s)' % \
+ (self._buf, self._parent_width, self._byte_width, self._type)
+
+ @property
+ def _Bytes(self):
+ return self._buf[:self._parent_width]
+
+ def _ConvertError(self, target_type):
+ raise TypeError('cannot convert %s to %s' % (self._type, target_type))
+
+ def _Indirect(self):
+ return self._buf.Indirect(0, self._parent_width)
+
+ @property
+ def IsNull(self):
+ return self._type is Type.NULL
+
+ @property
+ def IsBool(self):
+ return self._type is Type.BOOL
+
+ @property
+ def AsBool(self):
+ if self._type is Type.BOOL:
+ return bool(_Unpack(U, self._Bytes))
+ else:
+ return self.AsInt != 0
+
+ def MutateBool(self, value):
+ """Mutates underlying boolean value bytes in place.
+
+ Args:
+ value: New boolean value.
+
+ Returns:
+ Whether the value was mutated or not.
+ """
+ return self.IsBool and \
+ _Mutate(U, self._buf, value, self._parent_width, BitWidth.W8)
+
+ @property
+ def IsNumeric(self):
+ return self.IsInt or self.IsFloat
+
+ @property
+ def IsInt(self):
+ return self._type in (Type.INT, Type.INDIRECT_INT, Type.UINT,
+ Type.INDIRECT_UINT)
+
+ @property
+ def AsInt(self):
+ """Returns current reference as integer value."""
+ if self.IsNull:
+ return 0
+ elif self.IsBool:
+ return int(self.AsBool)
+ elif self._type is Type.INT:
+ return _Unpack(I, self._Bytes)
+ elif self._type is Type.INDIRECT_INT:
+ return _Unpack(I, self._Indirect()[:self._byte_width])
+ if self._type is Type.UINT:
+ return _Unpack(U, self._Bytes)
+ elif self._type is Type.INDIRECT_UINT:
+ return _Unpack(U, self._Indirect()[:self._byte_width])
+ elif self.IsString:
+ return len(self.AsString)
+ elif self.IsKey:
+ return len(self.AsKey)
+ elif self.IsBlob:
+ return len(self.AsBlob)
+ elif self.IsVector:
+ return len(self.AsVector)
+ elif self.IsTypedVector:
+ return len(self.AsTypedVector)
+ elif self.IsFixedTypedVector:
+ return len(self.AsFixedTypedVector)
+ else:
+ raise self._ConvertError(Type.INT)
+
+ def MutateInt(self, value):
+ """Mutates underlying integer value bytes in place.
+
+ Args:
+ value: New integer value. It must fit to the byte size of the existing
+ encoded value.
+
+ Returns:
+ Whether the value was mutated or not.
+ """
+ if self._type is Type.INT:
+ return _Mutate(I, self._buf, value, self._parent_width, BitWidth.I(value))
+ elif self._type is Type.INDIRECT_INT:
+ return _Mutate(I, self._Indirect(), value, self._byte_width,
+ BitWidth.I(value))
+ elif self._type is Type.UINT:
+ return _Mutate(U, self._buf, value, self._parent_width, BitWidth.U(value))
+ elif self._type is Type.INDIRECT_UINT:
+ return _Mutate(U, self._Indirect(), value, self._byte_width,
+ BitWidth.U(value))
+ else:
+ return False
+
+ @property
+ def IsFloat(self):
+ return self._type in (Type.FLOAT, Type.INDIRECT_FLOAT)
+
+ @property
+ def AsFloat(self):
+ """Returns current reference as floating point value."""
+ if self.IsNull:
+ return 0.0
+ elif self.IsBool:
+ return float(self.AsBool)
+ elif self.IsInt:
+ return float(self.AsInt)
+ elif self._type is Type.FLOAT:
+ return _Unpack(F, self._Bytes)
+ elif self._type is Type.INDIRECT_FLOAT:
+ return _Unpack(F, self._Indirect()[:self._byte_width])
+ elif self.IsString:
+ return float(self.AsString)
+ elif self.IsVector:
+ return float(len(self.AsVector))
+ elif self.IsTypedVector():
+ return float(len(self.AsTypedVector))
+ elif self.IsFixedTypedVector():
+ return float(len(self.FixedTypedVector))
+ else:
+ raise self._ConvertError(Type.FLOAT)
+
+ def MutateFloat(self, value):
+ """Mutates underlying floating point value bytes in place.
+
+ Args:
+ value: New float value. It must fit to the byte size of the existing
+ encoded value.
+
+ Returns:
+ Whether the value was mutated or not.
+ """
+ if self._type is Type.FLOAT:
+ return _Mutate(F, self._buf, value, self._parent_width,
+ BitWidth.B(self._parent_width))
+ elif self._type is Type.INDIRECT_FLOAT:
+ return _Mutate(F, self._Indirect(), value, self._byte_width,
+ BitWidth.B(self._byte_width))
+ else:
+ return False
+
+ @property
+ def IsKey(self):
+ return self._type is Type.KEY
+
+ @property
+ def AsKeyBytes(self):
+ if self.IsKey:
+ return Key(self._Indirect(), self._byte_width).Bytes
+ else:
+ raise self._ConvertError(Type.KEY)
+
+ @property
+ def AsKey(self):
+ if self.IsKey:
+ return str(Key(self._Indirect(), self._byte_width))
+ else:
+ raise self._ConvertError(Type.KEY)
+
+ @property
+ def IsString(self):
+ return self._type is Type.STRING
+
+ @property
+ def AsString(self):
+ if self.IsString:
+ return str(String(self._Indirect(), self._byte_width))
+ elif self.IsKey:
+ return self.AsKey
+ else:
+ raise self._ConvertError(Type.STRING)
+
+ def MutateString(self, value):
+ return String(self._Indirect(), self._byte_width).Mutate(value)
+
+ @property
+ def IsBlob(self):
+ return self._type is Type.BLOB
+
+ @property
+ def AsBlob(self):
+ if self.IsBlob:
+ return Blob(self._Indirect(), self._byte_width).Bytes
+ else:
+ raise self._ConvertError(Type.BLOB)
+
+ @property
+ def IsAnyVector(self):
+ return self.IsVector or self.IsTypedVector or self.IsFixedTypedVector()
+
+ @property
+ def IsVector(self):
+ return self._type in (Type.VECTOR, Type.MAP)
+
+ @property
+ def AsVector(self):
+ if self.IsVector:
+ return Vector(self._Indirect(), self._byte_width)
+ else:
+ raise self._ConvertError(Type.VECTOR)
+
+ @property
+ def IsTypedVector(self):
+ return Type.IsTypedVector(self._type)
+
+ @property
+ def AsTypedVector(self):
+ if self.IsTypedVector:
+ return TypedVector(self._Indirect(), self._byte_width,
+ Type.ToTypedVectorElementType(self._type))
+ else:
+ raise self._ConvertError('TYPED_VECTOR')
+
+ @property
+ def IsFixedTypedVector(self):
+ return Type.IsFixedTypedVector(self._type)
+
+ @property
+ def AsFixedTypedVector(self):
+ if self.IsFixedTypedVector:
+ element_type, size = Type.ToFixedTypedVectorElementType(self._type)
+ return TypedVector(self._Indirect(), self._byte_width, element_type, size)
+ else:
+ raise self._ConvertError('FIXED_TYPED_VECTOR')
+
+ @property
+ def IsMap(self):
+ return self._type is Type.MAP
+
+ @property
+ def AsMap(self):
+ if self.IsMap:
+ return Map(self._Indirect(), self._byte_width)
+ else:
+ raise self._ConvertError(Type.MAP)
+
+ @property
+ def Value(self):
+ """Converts current reference to value of corresponding type.
+
+ This is equivalent to calling `AsInt` for integer values, `AsFloat` for
+ floating point values, etc.
+
+ Returns:
+ Value of corresponding type.
+ """
+ if self.IsNull:
+ return None
+ elif self.IsBool:
+ return self.AsBool
+ elif self.IsInt:
+ return self.AsInt
+ elif self.IsFloat:
+ return self.AsFloat
+ elif self.IsString:
+ return self.AsString
+ elif self.IsKey:
+ return self.AsKey
+ elif self.IsBlob:
+ return self.AsBlob
+ elif self.IsMap:
+ return self.AsMap.Value
+ elif self.IsVector:
+ return self.AsVector.Value
+ elif self.IsTypedVector:
+ return self.AsTypedVector.Value
+ elif self.IsFixedTypedVector:
+ return self.AsFixedTypedVector.Value
+ else:
+ raise TypeError('cannot convert %r to value' % self)
+
+
+def _IsIterable(obj):
+ try:
+ iter(obj)
+ return True
+ except TypeError:
+ return False
+
+
+class Value:
+ """Class to represent given value during the encoding process."""
+
+ @staticmethod
+ def Null():
+ return Value(0, Type.NULL, BitWidth.W8)
+
+ @staticmethod
+ def Bool(value):
+ return Value(value, Type.BOOL, BitWidth.W8)
+
+ @staticmethod
+ def Int(value, bit_width):
+ return Value(value, Type.INT, bit_width)
+
+ @staticmethod
+ def UInt(value, bit_width):
+ return Value(value, Type.UINT, bit_width)
+
+ @staticmethod
+ def Float(value, bit_width):
+ return Value(value, Type.FLOAT, bit_width)
+
+ @staticmethod
+ def Key(offset):
+ return Value(offset, Type.KEY, BitWidth.W8)
+
+ def __init__(self, value, type_, min_bit_width):
+ self._value = value
+ self._type = type_
+
+ # For scalars: of itself, for vector: of its elements, for string: length.
+ self._min_bit_width = min_bit_width
+
+ @property
+ def Value(self):
+ return self._value
+
+ @property
+ def Type(self):
+ return self._type
+
+ @property
+ def MinBitWidth(self):
+ return self._min_bit_width
+
+ def StoredPackedType(self, parent_bit_width=BitWidth.W8):
+ return Type.Pack(self._type, self.StoredWidth(parent_bit_width))
+
+ # We have an absolute offset, but want to store a relative offset
+ # elem_index elements beyond the current buffer end. Since whether
+ # the relative offset fits in a certain byte_width depends on
+ # the size of the elements before it (and their alignment), we have
+ # to test for each size in turn.
+ def ElemWidth(self, buf_size, elem_index=0):
+ if Type.IsInline(self._type):
+ return self._min_bit_width
+ for byte_width in 1, 2, 4, 8:
+ offset_loc = buf_size + _PaddingBytes(buf_size, byte_width) + \
+ elem_index * byte_width
+ bit_width = BitWidth.U(offset_loc - self._value)
+ if byte_width == (1 << bit_width):
+ return bit_width
+ raise ValueError('relative offset is too big')
+
+ def StoredWidth(self, parent_bit_width=BitWidth.W8):
+ if Type.IsInline(self._type):
+ return max(self._min_bit_width, parent_bit_width)
+ return self._min_bit_width
+
+ def __repr__(self):
+ return 'Value(%s, %s, %s)' % (self._value, self._type, self._min_bit_width)
+
+ def __str__(self):
+ return str(self._value)
+
+
+def InMap(func):
+ def wrapper(self, *args, **kwargs):
+ if isinstance(args[0], str):
+ self.Key(args[0])
+ func(self, *args[1:], **kwargs)
+ else:
+ func(self, *args, **kwargs)
+ return wrapper
+
+
+def InMapForString(func):
+ def wrapper(self, *args):
+ if len(args) == 1:
+ func(self, args[0])
+ elif len(args) == 2:
+ self.Key(args[0])
+ func(self, args[1])
+ else:
+ raise ValueError('invalid number of arguments')
+ return wrapper
+
+
+class Pool:
+ """Collection of (data, offset) pairs sorted by data for quick access."""
+
+ def __init__(self):
+ self._pool = [] # sorted list of (data, offset) tuples
+
+ def FindOrInsert(self, data, offset):
+ do = data, offset
+ index = _BinarySearch(self._pool, do, lambda a, b: a[0] < b[0])
+ if index != -1:
+ _, offset = self._pool[index]
+ return offset
+ self._pool.insert(index, do)
+ return None
+
+ def Clear(self):
+ self._pool = []
+
+ @property
+ def Elements(self):
+ return [data for data, _ in self._pool]
+
+
+class Builder:
+ """Helper class to encode structural data into flexbuffers format."""
+
+ def __init__(self,
+ share_strings=False,
+ share_keys=True,
+ force_min_bit_width=BitWidth.W8):
+ self._share_strings = share_strings
+ self._share_keys = share_keys
+ self._force_min_bit_width = force_min_bit_width
+
+ self._string_pool = Pool()
+ self._key_pool = Pool()
+
+ self._finished = False
+ self._buf = bytearray()
+ self._stack = []
+
+ def __len__(self):
+ return len(self._buf)
+
+ @property
+ def StringPool(self):
+ return self._string_pool
+
+ @property
+ def KeyPool(self):
+ return self._key_pool
+
+ def Clear(self):
+ self._string_pool.Clear()
+ self._key_pool.Clear()
+ self._finished = False
+ self._buf = bytearray()
+ self._stack = []
+
+ def Finish(self):
+ """Finishes encoding process and returns underlying buffer."""
+ if self._finished:
+ raise RuntimeError('builder has been already finished')
+
+ # If you hit this exception, you likely have objects that were never
+ # included in a parent. You need to have exactly one root to finish a
+ # buffer. Check your Start/End calls are matched, and all objects are inside
+ # some other object.
+ if len(self._stack) != 1:
+ raise RuntimeError('internal stack size must be one')
+
+ value = self._stack[0]
+ byte_width = self._Align(value.ElemWidth(len(self._buf)))
+ self._WriteAny(value, byte_width=byte_width) # Root value
+ self._Write(U, value.StoredPackedType(), byte_width=1) # Root type
+ self._Write(U, byte_width, byte_width=1) # Root size
+
+ self.finished = True
+ return self._buf
+
+ def _ReadKey(self, offset):
+ key = self._buf[offset:]
+ return key[:key.find(0)]
+
+ def _Align(self, alignment):
+ byte_width = 1 << alignment
+ self._buf.extend(b'\x00' * _PaddingBytes(len(self._buf), byte_width))
+ return byte_width
+
+ def _Write(self, fmt, value, byte_width):
+ self._buf.extend(_Pack(fmt, value, byte_width))
+
+ def _WriteVector(self, fmt, values, byte_width):
+ self._buf.extend(_PackVector(fmt, values, byte_width))
+
+ def _WriteOffset(self, offset, byte_width):
+ relative_offset = len(self._buf) - offset
+ assert byte_width == 8 or relative_offset < (1 << (8 * byte_width))
+ self._Write(U, relative_offset, byte_width)
+
+ def _WriteAny(self, value, byte_width):
+ fmt = {
+ Type.NULL: U, Type.BOOL: U, Type.INT: I, Type.UINT: U, Type.FLOAT: F
+ }.get(value.Type)
+ if fmt:
+ self._Write(fmt, value.Value, byte_width)
+ else:
+ self._WriteOffset(value.Value, byte_width)
+
+ def _WriteBlob(self, data, append_zero, type_):
+ bit_width = BitWidth.U(len(data))
+ byte_width = self._Align(bit_width)
+ self._Write(U, len(data), byte_width)
+ loc = len(self._buf)
+ self._buf.extend(data)
+ if append_zero:
+ self._buf.append(0)
+ self._stack.append(Value(loc, type_, bit_width))
+ return loc
+
+ def _WriteScalarVector(self, element_type, byte_width, elements, fixed):
+ """Writes scalar vector elements to the underlying buffer."""
+ bit_width = BitWidth.B(byte_width)
+ # If you get this exception, you're trying to write a vector with a size
+ # field that is bigger than the scalars you're trying to write (e.g. a
+ # byte vector > 255 elements). For such types, write a "blob" instead.
+ if BitWidth.U(len(elements)) > bit_width:
+ raise ValueError('too many elements for the given byte_width')
+
+ self._Align(bit_width)
+ if not fixed:
+ self._Write(U, len(elements), byte_width)
+
+ loc = len(self._buf)
+
+ fmt = {Type.INT: I, Type.UINT: U, Type.FLOAT: F}.get(element_type)
+ if not fmt:
+ raise TypeError('unsupported element_type')
+ self._WriteVector(fmt, elements, byte_width)
+
+ type_ = Type.ToTypedVector(element_type, len(elements) if fixed else 0)
+ self._stack.append(Value(loc, type_, bit_width))
+ return loc
+
+ def _CreateVector(self, elements, typed, fixed, keys=None):
+ """Writes vector elements to the underlying buffer."""
+ length = len(elements)
+
+ if fixed and not typed:
+ raise ValueError('fixed vector must be typed')
+
+ # Figure out smallest bit width we can store this vector with.
+ bit_width = max(self._force_min_bit_width, BitWidth.U(length))
+ prefix_elems = 1 # Vector size
+ if keys:
+ bit_width = max(bit_width, keys.ElemWidth(len(self._buf)))
+ prefix_elems += 2 # Offset to the keys vector and its byte width.
+
+ vector_type = Type.KEY
+ # Check bit widths and types for all elements.
+ for i, e in enumerate(elements):
+ bit_width = max(bit_width, e.ElemWidth(len(self._buf), prefix_elems + i))
+
+ if typed:
+ if i == 0:
+ vector_type = e.Type
+ else:
+ if vector_type != e.Type:
+ raise RuntimeError('typed vector elements must be of the same type')
+
+ if fixed and not Type.IsFixedTypedVectorElementType(vector_type):
+ raise RuntimeError('must be fixed typed vector element type')
+
+ byte_width = self._Align(bit_width)
+ # Write vector. First the keys width/offset if available, and size.
+ if keys:
+ self._WriteOffset(keys.Value, byte_width)
+ self._Write(U, 1 << keys.MinBitWidth, byte_width)
+
+ if not fixed:
+ self._Write(U, length, byte_width)
+
+ # Then the actual data.
+ loc = len(self._buf)
+ for e in elements:
+ self._WriteAny(e, byte_width)
+
+ # Then the types.
+ if not typed:
+ for e in elements:
+ self._buf.append(e.StoredPackedType(bit_width))
+
+ if keys:
+ type_ = Type.MAP
+ else:
+ if typed:
+ type_ = Type.ToTypedVector(vector_type, length if fixed else 0)
+ else:
+ type_ = Type.VECTOR
+
+ return Value(loc, type_, bit_width)
+
+ def _PushIndirect(self, value, type_, bit_width):
+ byte_width = self._Align(bit_width)
+ loc = len(self._buf)
+ fmt = {
+ Type.INDIRECT_INT: I,
+ Type.INDIRECT_UINT: U,
+ Type.INDIRECT_FLOAT: F
+ }[type_]
+ self._Write(fmt, value, byte_width)
+ self._stack.append(Value(loc, type_, bit_width))
+
+ @InMapForString
+ def String(self, value):
+ """Encodes string value."""
+ reset_to = len(self._buf)
+ encoded = value.encode('utf-8')
+ loc = self._WriteBlob(encoded, append_zero=True, type_=Type.STRING)
+ if self._share_strings:
+ prev_loc = self._string_pool.FindOrInsert(encoded, loc)
+ if prev_loc is not None:
+ del self._buf[reset_to:]
+ self._stack[-1]._value = loc = prev_loc # pylint: disable=protected-access
+
+ return loc
+
+ @InMap
+ def Blob(self, value):
+ """Encodes binary blob value.
+
+ Args:
+ value: A byte/bytearray value to encode
+
+ Returns:
+ Offset of the encoded value in underlying the byte buffer.
+ """
+ return self._WriteBlob(value, append_zero=False, type_=Type.BLOB)
+
+ def Key(self, value):
+ """Encodes key value.
+
+ Args:
+ value: A byte/bytearray/str value to encode. Byte object must not contain
+ zero bytes. String object must be convertible to ASCII.
+
+ Returns:
+ Offset of the encoded value in the underlying byte buffer.
+ """
+ if isinstance(value, (bytes, bytearray)):
+ encoded = value
+ else:
+ encoded = value.encode('ascii')
+
+ if 0 in encoded:
+ raise ValueError('key contains zero byte')
+
+ loc = len(self._buf)
+ self._buf.extend(encoded)
+ self._buf.append(0)
+ if self._share_keys:
+ prev_loc = self._key_pool.FindOrInsert(encoded, loc)
+ if prev_loc is not None:
+ del self._buf[loc:]
+ loc = prev_loc
+
+ self._stack.append(Value.Key(loc))
+ return loc
+
+ def Null(self, key=None):
+ """Encodes None value."""
+ if key:
+ self.Key(key)
+ self._stack.append(Value.Null())
+
+ @InMap
+ def Bool(self, value):
+ """Encodes boolean value.
+
+ Args:
+ value: A boolean value.
+ """
+ self._stack.append(Value.Bool(value))
+
+ @InMap
+ def Int(self, value, byte_width=0):
+ """Encodes signed integer value.
+
+ Args:
+ value: A signed integer value.
+ byte_width: Number of bytes to use: 1, 2, 4, or 8.
+ """
+ bit_width = BitWidth.I(value) if byte_width == 0 else BitWidth.B(byte_width)
+ self._stack.append(Value.Int(value, bit_width))
+
+ @InMap
+ def IndirectInt(self, value, byte_width=0):
+ """Encodes signed integer value indirectly.
+
+ Args:
+ value: A signed integer value.
+ byte_width: Number of bytes to use: 1, 2, 4, or 8.
+ """
+ bit_width = BitWidth.I(value) if byte_width == 0 else BitWidth.B(byte_width)
+ self._PushIndirect(value, Type.INDIRECT_INT, bit_width)
+
+ @InMap
+ def UInt(self, value, byte_width=0):
+ """Encodes unsigned integer value.
+
+ Args:
+ value: An unsigned integer value.
+ byte_width: Number of bytes to use: 1, 2, 4, or 8.
+ """
+ bit_width = BitWidth.U(value) if byte_width == 0 else BitWidth.B(byte_width)
+ self._stack.append(Value.UInt(value, bit_width))
+
+ @InMap
+ def IndirectUInt(self, value, byte_width=0):
+ """Encodes unsigned integer value indirectly.
+
+ Args:
+ value: An unsigned integer value.
+ byte_width: Number of bytes to use: 1, 2, 4, or 8.
+ """
+ bit_width = BitWidth.U(value) if byte_width == 0 else BitWidth.B(byte_width)
+ self._PushIndirect(value, Type.INDIRECT_UINT, bit_width)
+
+ @InMap
+ def Float(self, value, byte_width=0):
+ """Encodes floating point value.
+
+ Args:
+ value: A floating point value.
+ byte_width: Number of bytes to use: 4 or 8.
+ """
+ bit_width = BitWidth.F(value) if byte_width == 0 else BitWidth.B(byte_width)
+ self._stack.append(Value.Float(value, bit_width))
+
+ @InMap
+ def IndirectFloat(self, value, byte_width=0):
+ """Encodes floating point value indirectly.
+
+ Args:
+ value: A floating point value.
+ byte_width: Number of bytes to use: 4 or 8.
+ """
+ bit_width = BitWidth.F(value) if byte_width == 0 else BitWidth.B(byte_width)
+ self._PushIndirect(value, Type.INDIRECT_FLOAT, bit_width)
+
+ def _StartVector(self):
+ """Starts vector construction."""
+ return len(self._stack)
+
+ def _EndVector(self, start, typed, fixed):
+ """Finishes vector construction by encodung its elements."""
+ vec = self._CreateVector(self._stack[start:], typed, fixed)
+ del self._stack[start:]
+ self._stack.append(vec)
+ return vec.Value
+
+ @contextlib.contextmanager
+ def Vector(self, key=None):
+ if key:
+ self.Key(key)
+
+ try:
+ start = self._StartVector()
+ yield self
+ finally:
+ self._EndVector(start, typed=False, fixed=False)
+
+ @InMap
+ def VectorFromElements(self, elements):
+ """Encodes sequence of any elements as a vector.
+
+ Args:
+ elements: sequence of elements, they may have different types.
+ """
+ with self.Vector():
+ for e in elements:
+ self.Add(e)
+
+ @contextlib.contextmanager
+ def TypedVector(self, key=None):
+ if key:
+ self.Key(key)
+
+ try:
+ start = self._StartVector()
+ yield self
+ finally:
+ self._EndVector(start, typed=True, fixed=False)
+
+ @InMap
+ def TypedVectorFromElements(self, elements, element_type=None):
+ """Encodes sequence of elements of the same type as typed vector.
+
+ Args:
+ elements: Sequence of elements, they must be of the same type.
+ element_type: Suggested element type. Setting it to None means determining
+ correct value automatically based on the given elements.
+ """
+ if isinstance(elements, array.array):
+ if elements.typecode == 'f':
+ self._WriteScalarVector(Type.FLOAT, 4, elements, fixed=False)
+ elif elements.typecode == 'd':
+ self._WriteScalarVector(Type.FLOAT, 8, elements, fixed=False)
+ elif elements.typecode in ('b', 'h', 'i', 'l', 'q'):
+ self._WriteScalarVector(
+ Type.INT, elements.itemsize, elements, fixed=False)
+ elif elements.typecode in ('B', 'H', 'I', 'L', 'Q'):
+ self._WriteScalarVector(
+ Type.UINT, elements.itemsize, elements, fixed=False)
+ else:
+ raise ValueError('unsupported array typecode: %s' % elements.typecode)
+ else:
+ add = self.Add if element_type is None else self.Adder(element_type)
+ with self.TypedVector():
+ for e in elements:
+ add(e)
+
+ @InMap
+ def FixedTypedVectorFromElements(self,
+ elements,
+ element_type=None,
+ byte_width=0):
+ """Encodes sequence of elements of the same type as fixed typed vector.
+
+ Args:
+ elements: Sequence of elements, they must be of the same type. Allowed
+ types are `Type.INT`, `Type.UINT`, `Type.FLOAT`. Allowed number of
+ elements are 2, 3, or 4.
+ element_type: Suggested element type. Setting it to None means determining
+ correct value automatically based on the given elements.
+ byte_width: Number of bytes to use per element. For `Type.INT` and
+ `Type.UINT`: 1, 2, 4, or 8. For `Type.FLOAT`: 4 or 8. Setting it to 0
+ means determining correct value automatically based on the given
+ elements.
+ """
+ if not 2 <= len(elements) <= 4:
+ raise ValueError('only 2, 3, or 4 elements are supported')
+
+ types = {type(e) for e in elements}
+ if len(types) != 1:
+ raise TypeError('all elements must be of the same type')
+
+ type_, = types
+
+ if element_type is None:
+ element_type = {int: Type.INT, float: Type.FLOAT}.get(type_)
+ if not element_type:
+ raise TypeError('unsupported element_type: %s' % type_)
+
+ if byte_width == 0:
+ width = {
+ Type.UINT: BitWidth.U,
+ Type.INT: BitWidth.I,
+ Type.FLOAT: BitWidth.F
+ }[element_type]
+ byte_width = 1 << max(width(e) for e in elements)
+
+ self._WriteScalarVector(element_type, byte_width, elements, fixed=True)
+
+ def _StartMap(self):
+ """Starts map construction."""
+ return len(self._stack)
+
+ def _EndMap(self, start):
+ """Finishes map construction by encodung its elements."""
+ # Interleaved keys and values on the stack.
+ stack = self._stack[start:]
+
+ if len(stack) % 2 != 0:
+ raise RuntimeError('must be even number of keys and values')
+
+ for key in stack[::2]:
+ if key.Type is not Type.KEY:
+ raise RuntimeError('all map keys must be of %s type' % Type.KEY)
+
+ pairs = zip(stack[::2], stack[1::2]) # [(key, value), ...]
+ pairs = sorted(pairs, key=lambda pair: self._ReadKey(pair[0].Value))
+
+ del self._stack[start:]
+ for pair in pairs:
+ self._stack.extend(pair)
+
+ keys = self._CreateVector(self._stack[start::2], typed=True, fixed=False)
+ values = self._CreateVector(
+ self._stack[start + 1::2], typed=False, fixed=False, keys=keys)
+
+ del self._stack[start:]
+ self._stack.append(values)
+ return values.Value
+
+ @contextlib.contextmanager
+ def Map(self, key=None):
+ if key:
+ self.Key(key)
+
+ try:
+ start = self._StartMap()
+ yield self
+ finally:
+ self._EndMap(start)
+
+ def MapFromElements(self, elements):
+ start = self._StartMap()
+ for k, v in elements.items():
+ self.Key(k)
+ self.Add(v)
+ self._EndMap(start)
+
+ def Adder(self, type_):
+ return {
+ Type.BOOL: self.Bool,
+ Type.INT: self.Int,
+ Type.INDIRECT_INT: self.IndirectInt,
+ Type.UINT: self.UInt,
+ Type.INDIRECT_UINT: self.IndirectUInt,
+ Type.FLOAT: self.Float,
+ Type.INDIRECT_FLOAT: self.IndirectFloat,
+ Type.KEY: self.Key,
+ Type.BLOB: self.Blob,
+ Type.STRING: self.String,
+ }[type_]
+
+ @InMapForString
+ def Add(self, value):
+ """Encodes value of any supported type."""
+ if value is None:
+ self.Null()
+ elif isinstance(value, bool):
+ self.Bool(value)
+ elif isinstance(value, int):
+ self.Int(value)
+ elif isinstance(value, float):
+ self.Float(value)
+ elif isinstance(value, str):
+ self.String(value)
+ elif isinstance(value, (bytes, bytearray)):
+ self.Blob(value)
+ elif isinstance(value, dict):
+ with self.Map():
+ for k, v in value.items():
+ self.Key(k)
+ self.Add(v)
+ elif isinstance(value, array.array):
+ self.TypedVectorFromElements(value)
+ elif _IsIterable(value):
+ self.VectorFromElements(value)
+ else:
+ raise TypeError('unsupported python type: %s' % type(value))
+
+ @property
+ def LastValue(self):
+ return self._stack[-1]
+
+ @InMap
+ def ReuseValue(self, value):
+ self._stack.append(value)
+
+
+def GetRoot(buf):
+ """Returns root `Ref` object for the given buffer."""
+ if len(buf) < 3:
+ raise ValueError('buffer is too small')
+ byte_width = buf[-1]
+ return Ref.PackedType(
+ Buf(buf, -(2 + byte_width)), byte_width, packed_type=buf[-2])
+
+
+def Dumps(obj):
+ """Returns bytearray with the encoded python object."""
+ fbb = Builder()
+ fbb.Add(obj)
+ return fbb.Finish()
+
+
+def Loads(buf):
+ """Returns python object decoded from the buffer."""
+ return GetRoot(buf).Value
diff --git a/python/setup.py b/python/setup.py
index b3fc566..0615f2b 100644
--- a/python/setup.py
+++ b/python/setup.py
@@ -12,18 +12,28 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import fileinput
import os
+import re
+import sys
from datetime import datetime
from setuptools import setup
+def _update_version_attr(new_version):
+ for line in fileinput.input('flatbuffers/_version.py', inplace=True):
+ if line.startswith('__version__'):
+ line = re.sub(r'".*"', '"{}"'.format(new_version), line)
+ sys.stdout.write(line)
+
+
def version():
version = os.getenv('VERSION', None)
if version:
# Most git tags are prefixed with 'v' (example: v1.2.3) this is
# never desirable for artifact repositories, so we strip the
# leading 'v' if it's present.
- return version[1:] if version.startswith('v') else version
+ version = version[1:] if version.startswith('v') else version
else:
# Default version is an ISO8601 compiliant datetime. PyPI doesn't allow
# the colon ':' character in its versions, and time is required to allow
@@ -39,8 +49,11 @@
print("VERSION environment variable not set, using datetime instead: {}"
.format(version))
+ _update_version_attr(version)
+
return version
+
setup(
name='flatbuffers',
version=version(),