🌟 Welcome to this week's update on Dozer! We are thrilled to announce the release of version v0.1.18, which brings a range of new features and improvements. Our team has been working diligently to enhance the overall functionality, and we are eager for you to experience all the new additions. Here are the updates for this week.
Key enhancement of v0.1.18 🚀
Support for Loading Dozer Config from a URL #1440 🌐:
In this release, Dozer introduces the ability to load configuration files from a remote URL. This feature simplifies configuration management and deployment. Now, you can run the following command to load the remore dozer config file:
dozer -c remote_url/dozer_config.yaml
Configurable Secondary Index #1461 🔧:
Secondary indexes can now be customized, offering more control over the database structure and optimization. To get gist of this feature, we will use a data sample from the following repository: https://github.com/getdozer/dozer-samples/tree/main/sql/join
Before implementing the secondary index configuration, the following dozer configuration would throw a 500 error:
connections:
- config : !LocalStorage
details:
path: data
tables:
- !Table
name: trips
prefix: /trips
file_type: parquet
extension: .parquet
name: ny_taxi
endpoints:
- name: trips
path: /trips
table_name: trips
index:
secondary:
create:
cache_max_map_size: 10000000000
query the data:
POST localhost:8080/trips/query
{
"$filter": {
"hvfhs_license_num": "HV0003"
},
"$order_by": {
"trip_miles": "desc"
}
}
the query result will throw an error:
Failed to query cache: Plan error: Matching index not found. Try to add following secondary index configuration:
- index: !SortedInverted
fields:
- hvfhs_license_num
- trip_miles
Now, you can create and configure the secondary index as follows:
g : !LocalStorage
details:
path: data
tables:
- !Table
name: trips
prefix: /trips
file_type: parquet
extension: .parquet
name: ny_taxi
endpoints:
- name: trips
path: /trips
table_name: trips
index:
secondary:
create:
- index: !SortedInverted
fields:
- hvfhs_license_num
- trip_miles
cache_max_map_size: 10000000000
After configuring the secondary index, you will get a query result like this:
[
{
"hvfhs_license_num": "HV0003",
"dispatching_base_num": "B03404",
"originating_base_num": "B03404",
"request_datetime": "2022-01-01T00:13:51.000Z",
"on_scene_datetime": "2022-01-01T00:19:52.000Z",
"pickup_datetime": "2022-01-01T00:19:52.000Z",
"dropoff_datetime": "2022-01-01T01:34:54.000Z",
"PULocationID": 138,
"DOLocationID": 265,
"trip_miles": 63.33,
"trip_time": 4502,
"base_passenger_fare": 166.66,
"tolls": 26.55,
"bcf": 5.87,
"sales_tax": 0.0,
"congestion_surcharge": 0.0,
"airport_fee": 2.5,
"tips": 0.0,
"driver_pay": 127.05,
"shared_request_flag": "N",
"shared_match_flag": "N",
"access_a_ride_flag": " ",
"wav_request_flag": "N",
"wav_match_flag": "N",
"__dozer_record_id": 8283,
"__dozer_record_version": 1
},
...
]
Other Fixes and Improvements 🛠️:
A variety of fixes and improvements have been made in this release, including UUID Postgres type support (#1458), optional AccessFilter::fields (#1436), and improved error messages for CustomTypeNotSupported (#1459).
New Contributor Spotlight 🌟
We would like to extend a warm welcome to @nurikk, who made their first contribution in #1460. We appreciate your support and look forward to your future contributions!
Dozer v0.1.18 is available now. Check out the release notes here.
Looking Forward & Spoiler Alert 🌈
As we continue to make strides in improving Dozer, we remain committed to enhancing its features and functionality. Your feedback is invaluable to us, so please don't hesitate to reach out with any suggestions or ideas. You can always open an issue for feature request OR start a discusisuon thread under our GitHub Q&A category. Together, we can make Dozer the best it can be!
Dozer Cloud will be out soon with rich set of features, enabling developers to to build and deploy Data APIs with more ease! ✨ 🧙♀️
Full Changelog
Contact us 📬
- GitHub: https://github.com/getdozer/dozer
- Discord: https://discord.com/invite/3eWXBgJaEQ
- Twitter: https://twitter.com/GetDozer
- LinkedIn: https://www.linkedin.com/company/getdozer/
