Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update macOS to 13.5 and xcode 14.3.1. #33722

Merged
merged 8 commits into from
Sep 14, 2023

Conversation

BillyONeal
Copy link
Member

Remove references to the sshfs stuff we no longer use. Change instructions to no longer need to install powershell or deploy a vcpkg clone on the hosts.

Remove references to the sshfs stuff we no longer use.
Change instructions to no longer need to install powershell or deploy a vcpkg clone on the hosts.
@BillyONeal BillyONeal added info:internal This PR or Issue was filed by the vcpkg team. category:infrastructure Pertaining to the CI/Testing infrastrucutre labels Sep 12, 2023
@Adela0814 Adela0814 self-assigned this Sep 12, 2023
Adela0814
Adela0814 previously approved these changes Sep 13, 2023
@Adela0814 Adela0814 added the info:reviewed Pull Request changes follow basic guidelines label Sep 13, 2023
This port has so much vendored stuff and is no longer maintained so I think it should probably be deindexed but this is enough for now.

```
/Users/vagrant/Data/buildtrees/urho3d/src/3eb4f765c8-520911ca5c.clean/Source/ThirdParty/SDL/src/file/cocoa/SDL_rwopsbundlesupport.m
/Users/vagrant/Data/buildtrees/urho3d/src/3eb4f765c8-520911ca5c.clean/Source/ThirdParty/SDL/src/file/cocoa/SDL_rwopsbundlesupport.m:45:20: error: mixing declarations and code is incompatible with standards before C99 [-Werror,-Wdeclaration-after-statement]
    NSFileManager* file_manager = [NSFileManager defaultManager];
                   ^
1 error generated.
```
@BillyONeal
Copy link
Member Author

REGRESSION: arrayfire:x64-osx failed with BUILD_FAILED. If expected, add arrayfire:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.

ld: warning: alignment (4) of atom '.2.4_2_kmpc_loc_struct_pack.12' is too small and may result in unaligned pointers 
ld: warning: alignment (4) of atom '.2.4_2_kmpc_loc_struct_pack.20' is too small and may result in unaligned pointers 
ld: warning: alignment (4) of atom '.2.4_2_kmpc_loc_struct_pack.28' is too small and may result in unaligned pointers 
ld: warning: pointer not aligned at page boundary address 0x9FA6FFC ('.2.190_2_kmpc_loc_struct_pack.357' + 16 from /Users/vagrant/Data/installed/x64-osx/lib/intel64/libmkl_intel_thread.a(s__gemm_drv.o))
ld: warning: pointer not aligned at page boundary address 0x9FB1FFC ('.2.47_2_kmpc_loc_struct_pack.20' + 16 from /Users/vagrant/Data/installed/x64-osx/lib/intel64/libmkl_intel_thread.a(ccsr0_smout_par.o))
ld: warning: pointer not aligned at page boundary address 0x9FC4FFC ('.2.64_2_kmpc_loc_struct_pack.12' + 16 from /Users/vagrant/Data/installed/x64-osx/lib/intel64/libmkl_intel_thread.a(zcoo0_mmout_par.o))
ld: warning: pointer not aligned at page boundary address 0x9FDFFFC ('.2.131_2_kmpc_loc_struct_pack.226' + 16 from /Users/vagrant/Data/installed/x64-osx/lib/intel64/libmkl_intel_thread.a(optimize_trsv_c_s_i8_omp.o))
ld: warning: pointer not aligned at page boundary address 0x9FEBFFC ('.2.9_2_kmpc_loc_struct_pack.76' + 16 from /Users/vagrant/Data/installed/x64-osx/lib/intel64/libmkl_intel_thread.a(xcsr__g_n_symgs_r_d_i8_omp.o))
ld: unaligned pointer(s) for architecture x86_64

This looks like some form of macos tooling bug. This port has also been extremely flaky in the past, I'm skipping it. Also relatively unpopular; installed 0.07% as frequently as zlib.

REGRESSION: blitz:x64-osx failed with BUILD_FAILED. If expected, add blitz:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.
REGRESSION: keystone:x64-osx failed with BUILD_FAILED. If expected, add keystone:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.

Python 2 is dead.

REGRESSION: libtorch:x64-osx failed with BUILD_FAILED. If expected, add libtorch:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.

/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/__algorithm/iterator_operations.h:100:5: error: static_assert failed due to requirement 'is_same<c10::impl::ListElementReference<at::Tensor, std::__wrap_iter<c10::IValue *>>, at::Tensor &>::value' "It looks like your iterator's `iterator_traits<It>::reference` does not match the return type of dereferencing the iterator, i.e., calling `*it`. This is undefined behavior according to [input.iterators] and can lead to dangling reference issues at runtime, so we are flagging this."
    static_assert(is_same<__deref_t<_Iter>, typename iterator_traits<__uncvref_t<_Iter> >::reference>::value,
    ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Looks fixed by pytorch/pytorch@9590cf6 ? Cherry-picked.

REGRESSION: coroutine:x64-osx failed with BUILD_FAILED. If expected, add coroutine:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.

Filed luncliff/coroutine#61 , baselined.

REGRESSION: tensorflow:x64-osx failed with BUILD_FAILED. If expected, add tensorflow:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.
REGRESSION: tensorflow-cc:x64-osx failed with BUILD_FAILED. If expected, add tensorflow-cc:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.

In file included from external/boringssl/err_data.c:18:
In file included from /usr/local/include/openssl/err.h:29:
In file included from /usr/local/include/openssl/bio.h:30:
/usr/local/include/openssl/crypto.h:308:3: error: typedef redefinition with different types ('struct crypto_threadid_st' vs 'int')
} CRYPTO_THREADID;
  ^
external/boringssl/src/include/openssl/base.h:318:13: note: previous definition is here
typedef int CRYPTO_THREADID;
            ^
external/boringssl/err_data.c:33:1: error: static_assert failed due to requirement '13 == 12' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_ASN1 == 12, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:34:1: error: static_assert failed due to requirement '14 == 13' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_CONF == 13, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:35:1: error: static_assert failed due to requirement '15 == 14' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_CRYPTO == 14, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:36:1: error: static_assert failed due to requirement '16 == 15' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_EC == 15, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:37:1: error: static_assert failed due to requirement '20 == 16' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_SSL == 16, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:38:1: error: static_assert failed due to requirement '32 == 17' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_BIO == 17, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:39:1: error: static_assert failed due to requirement '33 == 18' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_PKCS7 == 18, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:40:23: error: use of undeclared identifier 'ERR_LIB_PKCS8'
OPENSSL_STATIC_ASSERT(ERR_LIB_PKCS8 == 19, "library value changed");
                      ^
external/boringssl/err_data.c:41:1: error: static_assert failed due to requirement '34 == 20' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_X509V3 == 20, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:42:1: error: static_assert failed due to requirement '36 == 21' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_RAND == 21, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:43:1: error: static_assert failed due to requirement '38 == 22' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_ENGINE == 22, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:44:1: error: static_assert failed due to requirement '39 == 23' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_OCSP == 23, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:45:1: error: static_assert failed due to requirement '40 == 24' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_UI == 24, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
external/boringssl/err_data.c:46:1: error: static_assert failed due to requirement '41 == 25' "library value changed"
OPENSSL_STATIC_ASSERT(ERR_LIB_COMP == 25, "library value changed");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/boringssl/src/include/openssl/type_check.h:75:42: note: expanded from macro 'OPENSSL_STATIC_ASSERT'
#define OPENSSL_STATIC_ASSERT(cond, msg) _Static_assert(cond, msg)
                                         ^              ~~~~
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
INFO: Elapsed time: 275.192s, Critical Path: 18.05s
INFO: 2074 processes: 1169 internal, 905 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Tensorflow's attempt to force boringssl appears to now disagree with the system openssl. Likely this has always been broken, and Bazel is making it very hard to force tensorflow to use the copy of openssl from vcpkg. This isn't an infrastructure issue, and we've been talking about turning off tensorflow on osx for a while now as it takes 8 hours to build, so I'm baselining these.

REGRESSION: urho3d:x64-osx failed with BUILD_FAILED. If expected, add urho3d:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt.

Turns on a -Werror flag that is broken, patched out.

PASSING, REMOVE FROM FAIL LIST: bgfx:x64-osx (/Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt).
PASSING, REMOVE FROM FAIL LIST: concurrencpp:x64-osx (/Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt).

:)

@@ -2,7 +2,7 @@
"$note": "Due to the upstream modification of the dependent source code, the internal port of vcpkg cannot be used.",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this means 'urho3d cannot be in the curated registry at all' but that should be its own change.

@BillyONeal
Copy link
Member Author

I give up trying to ninja-patch libtorch, seems to be an understood problem upstream: pytorch/pytorch#100991

2.x update should fix

Comment on lines +1258 to +1261
tensorflow:x64-osx=skip
tensorflow:arm-osx=skip
tensorflow-cc:x64-osx=skip
tensorflow-cc:arm-osx=skip
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tensorflow ports are really bad. Building libs and tools for host and target, with many vendored dependencies. (More obvious for windows, where there are always some parts which seem to pick the wrong CRT.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah; I think there are some users of them on Windows or Linux where that we build from source effectively gives them -march=native which is a big perf reason to use us. But I don't think those folks have been looking to macos for a long time.

The 'google-y' projects just really hate working with anything that is not their internal build assumptions and it shows :(

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(We were considering turning off tensorflow builds on macos just due to the build time anyway so this was not a difficult decision)

@BillyONeal BillyONeal merged commit d99b693 into microsoft:master Sep 14, 2023
15 checks passed
@BillyONeal BillyONeal deleted the macos-update branch September 14, 2023 21:02
BillyONeal pushed a commit that referenced this pull request Apr 3, 2024
Passing on
https://dev.azure.com/vcpkg/public/_build/results?buildId=101160&view=results

Added `blitz` to ci.baseline.txt by #33722, which has been fixed by
#37723.

```
PASSING, REMOVE FROM FAIL LIST: blitz:x64-osx
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:infrastructure Pertaining to the CI/Testing infrastrucutre info:internal This PR or Issue was filed by the vcpkg team. info:reviewed Pull Request changes follow basic guidelines
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants