Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export document to jsonFile one by one #131

Open
wants to merge 6 commits into
base: v2
Choose a base branch
from
Open

Export document to jsonFile one by one #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1b905e6
write document one by one
letterbeezps Jun 20, 2023
517fd8c
add err check
letterbeezps Jun 20, 2023
2bb9d36
add close
letterbeezps Jun 20, 2023
fe14b77
export json : catch errors inside the function
letterbeezps Jun 21, 2023
6cd0197
fix conflict
letterbeezps Mar 14, 2024
fa7578e
capture export internal error
letterbeezps Mar 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ require (
github.com/google/orderedcode v0.0.1
github.com/klauspost/compress v1.17.0 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/stretchr/testify v1.8.4
github.com/vmihailenco/msgpack/v5 v5.3.5
Expand Down
44 changes: 31 additions & 13 deletions json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,38 +3,56 @@ package clover
import (
"bufio"
"encoding/json"
"fmt"
"os"

d "github.com/ostafen/clover/v2/document"
"github.com/ostafen/clover/v2/query"
)

// ExportCollection exports an existing collection to a JSON file.
func (db *DB) ExportCollection(collectionName string, exportPath string) error {
func (db *DB) ExportCollection(collectionName string, exportPath string) (err error) {
exists, err := db.HasCollection(collectionName)
if err != nil {
return err
}
if !exists {
return ErrCollectionNotExist
}

result, err := db.FindAll(query.NewQuery(collectionName))
q := query.NewQuery(collectionName)
f, err := os.Create(exportPath)
if err != nil {
return err
}
defer f.Close()

docs := make([]map[string]interface{}, 0)
for _, doc := range result {
docs = append(docs, doc.AsMap())
}

jsonString, err := json.Marshal(docs)
if err != nil {
return err
defer func() {
if p := recover(); p != nil {
err = fmt.Errorf("internal error: %v", p)
}
}()
isFirst := true
err = db.ForEach(q, func(doc *d.Document) bool {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Count() and the ForEach() statements run in separate transactions. Thus, if new documents are inserted between them, the ForEach() statement will iterate a number of documents n > collectionCount, which would lead to some problem with the statement:

if n != collectionCount {
	jsonString += ","
}

To prevent this issue, we can ensure the number of documents iterated by the ForEach() is exactly collectionCount using Limit():

err = db.ForEach(q.Limit(collectionCount), func(doc *d.Document) bool {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You remind me that is can also cause problems when documents are deleted between them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a suggestion for ForEach() function, let consumer() return error instead of bool. If there is any failure inside this function, we can get the error info directly

func (db *DB) ForEach(q *query.Query, consumer func(_ *d.Document) error) error {
	q, err := normalizeCriteria(q)
	if err != nil {
		return err
	}

	return db.IterateDocs(q, func(doc *d.Document) error {
		if err := consumer(doc); err != nil {
			return err
		}
		return nil
	})
}

jsonByte, err := json.Marshal(doc.AsMap())
if err != nil {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be a good idea to recover this error in the outer scope.
If there is any failure inside this function, then the export process would not report an error, but the result would be clearly not correct.

panic(err)
}
jsonString := string(jsonByte)
if isFirst {
isFirst = false
jsonString = "[" + jsonString
} else {
jsonString = "," + jsonString
}
if _, err := f.WriteString(jsonString); err != nil {
panic(err)
}
return true
})
if err == nil {
_, err = f.WriteString("]")
}

return os.WriteFile(exportPath, jsonString, os.ModePerm)
return
}

// ImportCollection imports a collection from a JSON file.
Expand Down
Loading