StringProperty value of StructuredProperty is coerced to bytes during choices validation → BadValueError

[blech75]

Expected Behavior

Given the following example ndb models and code:

from google.appengine.ext import ndb

RED = "red"
GREEN = "green"
BLUE = "blue"

COLORS = [RED, GREEN, BLUE]

class Car(ndb.Model):
    name = ndb.StringProperty(required=True)
    color = ndb.StringProperty(choices=COLORS, required=True)

class Dealership(ndb.Model):
    cars = ndb.StructuredProperty(Car, repeated=True)

d1 = Dealership(cars=[Car(name="Ferrari", color=RED)]).put().get()
print(d1)

d2 = Dealership.query(Dealership.cars == Car(color=RED)).get()
print(d2)

print(str(d1 == d2))
assert d1 == d2

...we should see:

Dealership(key=Key('Dealership', 1), cars=[Car(color='red', name='Ferrari')])
Dealership(key=Key('Dealership', 1), cars=[Car(color='red', name='Ferrari')])
True

...and there are no errors. This works fine in Python 2 with the Google Cloud SDK. 👍

Actual Behavior

Using Python 3.9.16 and appengine-python-standard, however, we see:

BadValueError: Value b'red' for property b'cars.color' is not an allowed choice`

...which is confusing because 'red' was provided as the value, not b'red'. 🤔🤔🤔

Digging around a bit, it appears to fail in StructuredProperty._comparison() (triggered by the Dealership.cars == Car(color=RED) expression), specifically right here:

appengine-python-standard/src/google/appengine/ext/ndb/model.py

Lines 2381 to 2382 in 882b8fa

	for prop in six.itervalues(self._modelclass._properties):
	vals = prop._get_base_value_unwrapped_as_list(value)

...because the provided color value is converted to a _BaseValue() of bytes via Property._get_base_value_unwrapped_as_list() → Property._get_base_value() → Property._opt_call_to_base_type() → Property._apply_to_values() → TextProperty._to_base_type():

appengine-python-standard/src/google/appengine/ext/ndb/model.py

Lines 1831 to 1833 in 882b8fa

	def _to_base_type(self, value):
	if isinstance(value, six.text_type):
	return value.encode('utf-8', 'surrogatepass')

...before being compared to the allowed choices (which are str). 😞

A workaround is to adjust the values in choices to be of type bytes:

RED = b"red"
GREEN = b"green"
BLUE = b"blue"

...and we then see the same result:

Dealership(key=Key('Dealership', 1), cars=[Car(color='red', name='Ferrari')])
Dealership(key=Key('Dealership', 1), cars=[Car(color='red', name='Ferrari')])
True

(This also works fine in Python 2 with the Google Cloud SDK. 😅)

Steps to Reproduce the Problem

(see code above)

Specifications

• Version: appengine-python-standard 1.1.3, Python 3.9.16
• Platform: MacOS (Darwin Kernel Version 23.0.0: Fri Sep 15 14:42:42 PDT 2023; root:xnu-10002.1.13~1/RELEASE_X86_64 x86_64)

---

Additional Info

This does not appear to have anything to do with persisting data -- the persisted value of Dealership.cars[].color remains a str: type(d1.cars[0].color) == str.

It only happens during "comparison" (a key part of querying) when the model with a StringProperty(choices=...) is a StructuredProperty of another model. . We can see this by skipping entity creation and just calling Dealership.query(Dealership.cars == Car(color=BLUE)), or even Dealership.cars == Car(color=BLUE) as the most minimal case.

When using a standalone model, all works fine:

ferrari = Car(name="Ferrari", color=RED).put().get()
print(ferrari)

red_car = Car.query(Car.color == RED).get()
print(red_car)

print(str(ferrari == red_car))
assert ferrari == red_car

...yields:

Car(key=Key('Car', 1), color='red', name='Ferrari')
Car(key=Key('Car', 1), color='red', name='Ferrari')
True