Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(go/adbc/driver): add support for Google BigQuery #1722

Merged
merged 67 commits into from
Jul 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
27d2f00
feat(go/driver/bigquery): basic skeletons
cocoa-xu Apr 15, 2024
6775bb0
feat(go/driver/bigquery): handled most used options in `bigquery.Quer…
cocoa-xu Apr 15, 2024
7229e98
feat(go/driver/bigquery): implemented `arrow.RecordReader`
cocoa-xu Apr 15, 2024
0755ac2
feat(go/driver/bigquery): support specifying credentials json filepath
cocoa-xu Apr 16, 2024
66a69da
feat(go/driver/bigquery): minor changes in `newRecordReader`
cocoa-xu Apr 16, 2024
542b748
feat(go/driver/bigquery): fixed an issue in `newClient`
cocoa-xu Apr 16, 2024
e66ff43
implements `driverbase.CurrentNamespacer`
cocoa-xu Apr 18, 2024
11d1e54
address issues in code review
cocoa-xu Apr 20, 2024
c36b16b
address issues in code review
cocoa-xu Apr 20, 2024
25a611a
implemented `GetTableSchema`
cocoa-xu Apr 21, 2024
389a72f
implemented python bindings
cocoa-xu Apr 21, 2024
fa6cdd8
implemented `GetTableSchema`
cocoa-xu Apr 21, 2024
ed0cfb0
implemented `GetObjectsCatalogs` and `GetObjectsDbSchemas`
cocoa-xu Apr 21, 2024
76a93fc
fixed memory issue in `GetTableSchema`
cocoa-xu Apr 21, 2024
c3fa9d9
implemented `driverbase.DbObjectsEnumerator`
cocoa-xu Apr 21, 2024
b6f5c63
fix `buildField` and `parsePrecisionAndScale`
cocoa-xu Apr 21, 2024
88a1c83
added support for using JSON credential string and access token
cocoa-xu Apr 21, 2024
521287a
go mod tidy
cocoa-xu Apr 24, 2024
ba24e7b
make `BigQueryTokenResponse` and handlers private
cocoa-xu Apr 24, 2024
b63a7f5
`SetAutocommit(true)` should be valid
cocoa-xu Apr 24, 2024
ffcd5a4
fix: close bigquery client in `connectionImpl.Close`
cocoa-xu Apr 24, 2024
ac92627
return `adbc.StatusNotImplemented` for `GetObjectsCatalogs`
cocoa-xu Apr 24, 2024
2690210
use `map[string]arrow.DataType` for simple data types
cocoa-xu Apr 24, 2024
0468d78
always store original `DATA_TYPE` value in metadata
cocoa-xu Apr 24, 2024
e8a31e1
remove implementation for `driverbase.DbObjectsEnumerator`
cocoa-xu Apr 24, 2024
cc7bf5a
fix: call `arrow.ListOf` in `buildField` for `ARRAY` data type
cocoa-xu Apr 24, 2024
f892fde
fix: get table types from bigquery directly
cocoa-xu Apr 24, 2024
0d98b4e
fix: use regexp for `parsePrecisionAndScale`
cocoa-xu Apr 24, 2024
fc375c8
fix: do not sanitize user inputs
cocoa-xu Apr 25, 2024
e06a12f
minor fix for comments
cocoa-xu Apr 28, 2024
a2e47a2
sanitize dataset name
cocoa-xu Apr 28, 2024
9246f1d
added basic support for Bind and BindStream
cocoa-xu May 5, 2024
04b5a07
Merge branch 'main' into feat/go-google-bigquery-support
cocoa-xu May 8, 2024
816e3a1
updated to use `github.com/apache/arrow/go/v17/*`
cocoa-xu May 8, 2024
a58803b
go mod tidy
cocoa-xu May 8, 2024
c88b37f
Merge branch 'main' into feat/go-google-bigquery-support
cocoa-xu May 22, 2024
0d50822
addressed issues mentioned in code review
cocoa-xu May 22, 2024
fb5b5d7
allow user to configure result record buffer size
cocoa-xu May 22, 2024
58d664b
addressed issues in code review
cocoa-xu Jun 1, 2024
4187737
Merge branch 'apache:main' into feat/go-google-bigquery-support
cocoa-xu Jun 1, 2024
29231b3
addressed issues in code review
cocoa-xu Jun 1, 2024
ccd22fc
allow user to configure prefetch concurrency
cocoa-xu Jun 1, 2024
0a096af
added very basic tests for bigquery
cocoa-xu Jun 12, 2024
defecf5
Merge branch 'main' into feat/go-google-bigquery-support
cocoa-xu Jun 12, 2024
986b64d
tests: added tests in c for bigquery
cocoa-xu Jun 12, 2024
3583e64
tests: fix memory leaks and added test for bulk-insert
cocoa-xu Jun 13, 2024
53b5fe4
tests: added `TestSqlIngestTimestampTypes`
cocoa-xu Jun 13, 2024
6a2bf56
tests: added `TestSqlIngestDate64Type`
cocoa-xu Jun 13, 2024
cb2f3e1
tests: added `TestSqlIngestDecimal`
cocoa-xu Jun 13, 2024
5e5b227
tests: added `TestNewDatabaseGetSetOptions`
cocoa-xu Jun 13, 2024
a9457a7
feat: added support for arrow.LIST types
cocoa-xu Jun 13, 2024
2e19e69
tests: added `TestSqlIngestListType`
cocoa-xu Jun 13, 2024
aad8948
update arrowValueToQueryParameterValue to cover all list variants
cocoa-xu Jun 13, 2024
ed6b109
feat: added support for arrow.STRUCT types
cocoa-xu Jun 13, 2024
ad83209
tests: added `TestSqlIngestStructType`
cocoa-xu Jun 13, 2024
9505865
tests: also test decimal256 in `TestSqlIngestDecimal`
cocoa-xu Jun 13, 2024
b2051cb
updated comments regarding unimplemented types
cocoa-xu Jun 13, 2024
5a884b7
Merge branch 'main' into feat/go-google-bigquery-support
cocoa-xu Jun 19, 2024
3fa30df
Merge branch 'main' into feat/go-google-bigquery-support
cocoa-xu Jun 22, 2024
05b54d4
go mod tidy
cocoa-xu Jun 22, 2024
7264efe
Merge branch 'main' into feat/go-google-bigquery-support
cocoa-xu Jun 27, 2024
3b9f699
address issues in code review
cocoa-xu Jul 1, 2024
72a8c3a
use `TableMetadata` in `getTableSchemaWithFilter`
cocoa-xu Jul 2, 2024
fee71e8
addressed issues in code review
cocoa-xu Jul 2, 2024
32c9200
addressed issues in code review
cocoa-xu Jul 3, 2024
7509398
addressed issues in code review
cocoa-xu Jul 3, 2024
060ed97
run queries and return records without parallelism
cocoa-xu Jul 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions c/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ if(ADBC_DRIVER_SNOWFLAKE)
add_subdirectory(driver/snowflake)
endif()

if(ADBC_DRIVER_BIGQUERY)
add_subdirectory(driver/bigquery)
endif()

if(ADBC_INTEGRATION_DUCKDB)
add_subdirectory(integration/duckdb)
endif()
Expand Down Expand Up @@ -118,6 +122,10 @@ LIBRARY=$<TARGET_FILE:adbc_driver_${TARGET}_shared>" ${Python3_EXECUTABLE} -m pi
if(ADBC_DRIVER_SNOWFLAKE)
adbc_install_python_package(snowflake)
endif()

if(ADBC_DRIVER_BIGQUERY)
adbc_install_python_package(bigquery)
endif()
endif()

validate_config()
Expand Down
1 change: 1 addition & 0 deletions c/cmake_modules/DefineOptions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
define_option(ADBC_DRIVER_POSTGRESQL "Build the PostgreSQL driver" OFF)
define_option(ADBC_DRIVER_SQLITE "Build the SQLite driver" OFF)
define_option(ADBC_DRIVER_SNOWFLAKE "Build the Snowflake driver" OFF)
define_option(ADBC_DRIVER_BIGQUERY "Build the BigQuery driver" OFF)

define_option(ADBC_INTEGRATION_DUCKDB "Build the test suite for DuckDB" OFF)
endif()
Expand Down
70 changes: 70 additions & 0 deletions c/driver/bigquery/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

include(GoUtils)

set(LDFLAGS "$<$<CONFIG:Release>:-s> $<$<CONFIG:Release>:-w>")
add_go_lib("${REPOSITORY_ROOT}/go/adbc/pkg/bigquery/"
adbc_driver_bigquery
SOURCES
driver.go
utils.h
utils.c
BUILD_TAGS
driverlib
PKG_CONFIG_NAME
adbc-driver-bigquery
SHARED_LINK_FLAGS
${LDFLAGS}
OUTPUTS
ADBC_LIBRARIES)

foreach(LIB_TARGET ${ADBC_LIBRARIES})
target_include_directories(${LIB_TARGET} SYSTEM
INTERFACE ${REPOSITORY_ROOT} ${REPOSITORY_ROOT}/c/
${REPOSITORY_ROOT}/c/vendor
${REPOSITORY_ROOT}/c/driver)
endforeach()

if(ADBC_TEST_LINKAGE STREQUAL "shared")
set(TEST_LINK_LIBS adbc_driver_bigquery_shared)
else()
set(TEST_LINK_LIBS adbc_driver_bigquery_static)
endif()

if(ADBC_BUILD_TESTS)
add_test_case(driver_bigquery_test
PREFIX
adbc
EXTRA_LABELS
driver-bigquery
SOURCES
bigquery_test.cc
EXTRA_LINK_LIBS
adbc_driver_common
adbc_validation
nanoarrow
${TEST_LINK_LIBS})
target_compile_features(adbc-driver-bigquery-test PRIVATE cxx_std_17)
target_include_directories(adbc-driver-bigquery-test SYSTEM
PRIVATE ${REPOSITORY_ROOT}
${REPOSITORY_ROOT}/c/
${REPOSITORY_ROOT}/c/vendor
${REPOSITORY_ROOT}/c/driver
${REPOSITORY_ROOT}/c/driver/common)
adbc_configure_target(adbc-driver-bigquery-test)
endif()
65 changes: 65 additions & 0 deletions c/driver/bigquery/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# ADBC Snowflake Driver

This driver provides an interface to
[BigQuery](https://cloud.google.com/bigquery) using ADBC.

## Building

See [CONTRIBUTING.md](../../CONTRIBUTING.md) for details.

## Testing

BigQuery credentials and project ID are required.

### Environment Variables
#### Project ID
Set `BIGQUERY_PROJECT_ID` to the project ID.

#### Authentication
Set either one following environment variables for authentication:

##### BIGQUERY_JSON_CREDENTIAL_FILE
Path to the JSON credential file. This file can be generated using `gcloud`:

```sh
gcloud auth application-default login
```

And the default location of the generated JSON credential file is located at

```sh
$HOME/.config/gcloud/application_default_credentials.json
```

##### BIGQUERY_JSON_CREDENTIAL_STRING
Store the whole JSON credential content, something like

```json
{
"account": "",
"client_id": "123456789012-1234567890abcdefabcdefabcdefabcd.apps.googleusercontent.com",
"client_secret": "d-SECRETSECRETSECRETSECR",
"refresh_token": "1//1234567890abcdefabcdefabcdef-abcdefabcd-abcdefabcdefabcdefabcdefab-abcdefabcdefabcdefabcdefabcdef-ab",
"type": "authorized_user",
"universe_domain": "googleapis.com"
}
```
25 changes: 25 additions & 0 deletions c/driver/bigquery/adbc-driver-bigquery.pc.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

prefix=@CMAKE_INSTALL_PREFIX@
libdir=@ADBC_PKG_CONFIG_LIBDIR@

Name: Apache Arrow Database Connectivity (ADBC) BigQuery driver
Description: The ADBC BigQuery driver provides an ADBC driver for BigQuery.
URL: https://github.com/apache/arrow-adbc
Version: @ADBC_VERSION@
Libs: -L${libdir} -ladbc_driver_bigquery
Loading
Loading