FLIP includes several benchmark datasets that contain a variety of protein sequences, each with a real-valued label indicating its "fitness" (how well the protein performs some particular function). The goal is to predict the fitness of a given protein sequence using the sequence. Different representations of protein sequences (e.g. learned embeddings from large language models) may prove helpful here. Some of the benchmark datasets (thermostability) contain a highly diverse set of sequences from many different protein families. Others (AAV, GB1) contain all sequences that are mutants of a single parent sequence. Each benchmark dataset features multiple "splits" -- different ways of train-test splitting the data to assess how well a model might generalize given limited information. The AAV benchmark, for example, features the "mutant vs designed" split in which a model is trained on randomly generated mutants and asked to predict the fitness of designed sequences, and the "seven vs many" split in which a model is trained on sequences with seven mutations and asked to make predictions for sequences with a different number of mutations.
The CATH (Class, Architecture, Topology, Homology) [65] database is a comprehensive resource for protein structure classification that hierarchical group proteins based on their structural features. The database defines classes based on topological similarities, architectures based on the arrangement of secondary structure elements, topologies based on the connectivity of secondary structure elements, and homologous domains based on sequence similarity.
The CATH (Class, Architecture, Topology, Homology) [65] database is a comprehensive resource for protein structure classification that hierarchical group proteins based on their structural features. The database defines classes based on topological similarities, architectures based on the arrangement of secondary structure elements, topologies based on the connectivity of secondary structure elements, and homologous domains based on sequence similarity. This results in a training set of 16,153 structures, a validation set of 1,457 structures, and a test set of 1,797 structures. Note that the curated CATH dataset contains only single-chain structures and does not consider the case of designing multi-chain proteins.
FLIP includes several benchmark datasets that contain a variety of protein sequences, each with a real-valued label indicating its "fitness" (how well the protein performs some particular function). The goal is to predict the fitness of a given protein sequence using the sequence. Different representations of protein sequences (e.g. learned embeddings from large language models) may prove helpful here. This sub-dataset (AAV) is a set of 201,426 training sequences and 82,583 test sequences in which the goal is to predict the fitness of mutants of the capsid protein from the adeno-associated virus (AAV). The training set proteins were designed, while the test set proteins are random mutants. The absolute value of the fitness is not important, but its ranking / relative value is -- protein designers would like to be able to pick a sequence with high fitness relative to those in the training set. Performance is therefore usually assessed using Spearman's r correlation coefficient.
Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski, Gerardo Tauriello, and Torsten Schwede.Continuous automated model evaluation (cameo)—perspectives on the future of fully automated evaluation of structure prediction methods.Proteins: Structure, Function, and Bioinformatics, 89:1977–1986, 12 2021.ISSN 0887-3585.doi: 10.1002/prot.26213.