Add files using upload-large-folder tool

.gitignore

@@ -0,0 +1 @@
+*.safetensors

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Meituan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in 
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,151 @@
----
-license: mit
----
+---
+license: mit
+library_name: LongCat-Flash-Lite
+pipeline_tag: text-generation
+tags:
+- transformers
+---
+
+# LongCat-Flash-Lite
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/meituan-longcat/LongCat-Flash-Chat/main/figures/longcat_logo.svg" 
+       width="300" 
+       alt="LongCat Logo"/>
+</div>
+
+<hr>
+
+
+<div align="center" style="line-height: 1;">
+
+  <a href="https://longcat.ai/" target="_blank" style="margin: 2px;">
+    <img alt="Chat" src="https://img.shields.io/badge/🤖%20Chat-LongCat--Flash--Chat-ADFF2F?color=29E154&logoColor=white"  fill-opacity="1" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+  <a href="https://huggingface.co/meituan-longcat" target="_blank" style="margin: 2px;">
+    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-LongCat-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+
+<div align="center" style="line-height: 1;">
+  <a href="https://github.com/meituan-longcat/LongCat-Flash-Chat/blob/main/figures/wechat_official_accounts.png" target="_blank" style="margin: 2px;">
+    <img alt="Wechat" src="https://img.shields.io/badge/WeChat-LongCat-brightgreen?logo=wechat&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+  <a href="https://x.com/Meituan_LongCat" target="_blank" style="margin: 2px;">
+    <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-LongCat-white?logo=x&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+
+<div align="center" style="line-height: 1;">
+  <a href="https://huggingface.co/meituan-longcat/LongCat-Flash-Chat/blob/main/LICENSE" style="margin: 2px;">
+    <img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+
+<p align="center">
+  <a href="https://xx.xx.xx"><b>Tech Report</b>&nbsp;📄</a>
+</p>
+
+
+## Model Introduction
+We introduce LongCat-Flash-Lite, a 68.5B parameter model with ∼3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.
+
+
+
+### Key Features
+
+#### 🌟  Superior Scaling Efficiency: A Better Alternative to MoE
+Through comprehensive scaling experiments across diverse scenarios, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared toincrease expert numbers, offering a high-efficiency alternative for model scaling.
+
+
+#### 🌟 Optimized Architecture with N-gram Embeddings
+We establish the complete set of architectural factors determining
+embedding scaling efficacy, covering the integration timing, parameter budgeting, hash collisions, hyperparameter
+settings and initialization of embedding, together with the effects of model width and depth. Besides, we investigate
+different methods of scaling embedding and find that N-gram Embedding offers the most robust scalability.
+
+
+#### 🌟 Inference Efficiency and System Optimization
+We demonstrate that N-gram Embedding largely reduce I/O bottlenecks in MoE layers, particularly when paired with speculative decoding to maximize hardware utilization. Addressing the concomitant embedding overhead, we propose a specialized N-gram Cache and synchronized kernels, ensuring that the reduction in active parameters translates directly to lower latency and higher throughput.
+
+For more detail, please refer to [***Scaling Embeddings Outperforms Scaling Experts in Language Models***](https://xx.xx.xx).
+
+## Evaluation Results
+Values marked with * are sourced from public reports.
+
+| Benchmark | Qwen3-Next-80B-A3B-Instruct | Qwen3-30B-A3B-Instruct-2507 | Gemini 2.5 Flash-Lite Preview (09-2025) | Kimi-Linear-48B-A3B | LongCat |
+|---------|-----------------------------|----------------------------|----------------------------------------|---------------------|---------|
+| **Architecture** | MoE | MoE | - | MoE | MoE + NE |
+| **# Total Params** | 80B | 30B | - | 48B | 68.5B |
+| **# Activated Params** | 3B | 3B | - | 3B | 2.9B∼4.5B |
+
+## Agentic Tool Use
+
+| Benchmark | Qwen3-Next-80B-A3B-Instruct | Qwen3-30B-A3B-Instruct-2507 | Gemini 2.5 Flash-Lite Preview (09-2025) | Kimi-Linear-48B-A3B | LongCat |
+|---------|-----------------------------|----------------------------|----------------------------------------|---------------------|---------|
+| Tau2-Airline(avg@8) | 45.5* | 38.0* | 35.00 | 44.00 | **58.00** |
+| Tau2-Retail(avg@8) | 57.3* | 57.0* | 37.50 | 18.86 | **73.10** |
+| Tau2-Telecom(avg@8) | 13.2* | 12.3* | 41.0* | 15.68 | **72.80** |
+| VitaBench(avg@4) | 5.80 | 3.50 | 4.50 | - | **7.00** |
+
+## Agentic Coding
+
+| Benchmark | Qwen3-Next-80B-A3B-Instruct | Qwen3-30B-A3B-Instruct-2507 | Gemini 2.5 Flash-Lite Preview (09-2025) | Kimi-Linear-48B-A3B | LongCat |
+|---------|-----------------------------|----------------------------|----------------------------------------|---------------------|---------|
+| SWE-Bench(acc) | 37.60 | 6.80 | 41.3* | 32.80 | **54.40** |
+| TerminalBench(acc) | 15.19 | 17.72 | 20.00 | 20.00 | **33.75** |
+| SWE-Bench Multiligual | 31.30 | 30.60 | - | 37.20 |  |
+
+## General Domains
+
+| Benchmark | Qwen3-Next-80B-A3B-Instruct | Qwen3-30B-A3B-Instruct-2507 | Gemini 2.5 Flash-Lite Preview (09-2025) | Kimi-Linear-48B-A3B | LongCat |
+|---------|-----------------------------|----------------------------|----------------------------------------|---------------------|---------|
+| GPQA-Diamond(avg@16) | **74.33** | 66.86 | 70.20* | 69.89 | 66.78 |
+| MMLU(acc) | **89.28** | 86.92 | 84.68 | 79.91 | 85.52 |
+| MMLU-Pro(acc) | **82.93** | 80.28 | 78.95 | 67.22 | 78.29 |
+| CEval(acc) | **90.91** | 90.05 | 75.16 | 78.48 | 86.55 |
+| CMMLU(acc) | **86.50** | 84.98 | 72.06 | 76.26 | 82.48 |
+
+## Mathematical Reasoning
+
+| Benchmark | Qwen3-Next-80B-A3B-Instruct | Qwen3-30B-A3B-Instruct-2507 | Gemini 2.5 Flash-Lite Preview (09-2025) | Kimi-Linear-48B-A3B | LongCat |
+|---------|-----------------------------|----------------------------|----------------------------------------|---------------------|---------|
+| MATH500(acc) | **98.00** | 97.00 | 95.20 | 94.20 | 96.80 |
+| AIME24(avg@32) | **81.35** | 71.35 | 63.33 | 70.52 | 72.19 |
+| AIME25(avg@32) | **68.44** | 62.08 | 50.1* | 59.58 | 63.23 |
+
+## Quick Start
+```shell
+
+```
+
+
+## License Agreement
+
+This repository, including both the model weights and the source code, is released under the **MIT License**. 
+
+Any contributions to this repository are licensed under the MIT License, unless otherwise stated. This license does not grant any rights to use Meituan trademarks or patents. 
+
+For details, see the [LICENSE](./LICENSE) file. 
+
+## Usage Considerations 
+This model has not been specifically designed or comprehensively evaluated for every possible downstream application. 
+
+Developers should take into account the known limitations of large language models, including performance variations across different languages, and carefully assess accuracy, safety, and fairness before deploying the model in sensitive or high-risk scenarios. 
+It is the responsibility of developers and downstream users to understand and comply with all applicable laws and regulations relevant to their use case, including but not limited to data protection, privacy, and content safety requirements. 
+
+Nothing in this Model Card should be interpreted as altering or restricting the terms of the MIT License under which the model is released. 
+
+
+## Citation
+
+We kindly encourage citation of our work if you find it useful.
+
+```
+
+```
+
+
+## Contact
+Please contact us at <a href="mailto:longcat-team@meituan.com">longcat-team@meituan.com</a> or open an issue if you have any questions.
+

config.json

@@ -0,0 +1,51 @@
+{
+  "architectures": [
+    "LongcatFlashNgramForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_longcat_ngram.LongcatFlashNgramConfig",
+    "AutoModel": "modeling_longcat_ngram.LongcatFlashNgramModel",
+    "AutoModelForCausalLM": "modeling_longcat_ngram.LongcatFlashNgramForCausalLM"
+  },
+  "vocab_size": 131072,
+  "hidden_size": 3072,
+  "ffn_hidden_size": 6144,
+  "expert_ffn_hidden_size": 1024,
+  "num_layers": 14,
+  "num_attention_heads": 32,
+  "kv_lora_rank": 512,
+  "q_lora_rank": 1536,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "qk_nope_head_dim": 128,
+  "mla_scale_q_lora": true,
+  "mla_scale_kv_lora": true,
+  "routed_scaling_factor": 6.0,
+  "n_routed_experts": 256,
+  "rms_norm_eps": 1e-5,
+  "use_cache": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "rope_theta": 5000000.0,
+  "max_position_embeddings": 327680,
+  "rope_scaling": {
+      "original_max_position_embeddings": 32768,
+      "rope_type": "yarn",
+      "factor": 10,
+      "beta_fast": 32,
+      "beta_slow": 1,
+      "mscale": 1,
+      "mscale_all_dim": 1
+  },
+  "zero_expert_num": 128,
+  "zero_expert_type": "identity",
+  "moe_topk": 12,
+  "ngram_vocab_size_ratio": 78,
+  "emb_neighbor_num": 4,
+  "emb_split_num": 4,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.6"
+}
+

configuration_longcat_ngram.py

@@ -0,0 +1,216 @@
+from transformers.models.longcat_flash import LongcatFlashConfig
+
+
+class LongcatFlashNgramConfig(LongcatFlashConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`LongcatFlashNgramModel`]. It is used to instantiate
+    a LongCat Flash model with N-gram enhanced embeddings according to the specified arguments, defining the model architecture. 
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 131072):
+            Vocabulary size of the LongCat Flash model. Defines the number of different tokens that can be represented by the
+            `input_ids` passed when calling [`LongcatFlashNgramModel`]
+        hidden_size (`int`, *optional*, defaults to 6144):
+            Dimension of the hidden representations.
+        num_hidden_layers (`int`, *optional*, defaults to 56):
+            Number of hidden layers in the Transformer decoder.
+        num_layers (`int`, *optional*, defaults to 28):
+            Number of layers, each with 2 sublayers.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        num_key_value_heads (`int`, *optional*):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting from a multi-head checkpoint to a GQA checkpoint, each group key and value head should be
+            constructed by meanpooling all the original heads within that group. For more details checkout [this
+            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
+            `num_attention_heads`.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 131072):
+            The maximum sequence length that this model might ever be used with. Typically set this to something large
+            just in case (e.g., 512 or 1024 or 2048).
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
+            The epsilon value used by the RMS normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            Padding token id.
+        bos_token_id (`int`, *optional*, defaults to 1):
+            Beginning of stream token id.
+        eos_token_id (`int`, *optional*, defaults to 2):
+            End of stream token id.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie input and output embeddings.
+        rope_theta (`float`, *optional*, defaults to 10000000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`.
+        attention_bias (`bool`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        ffn_hidden_size (`int`, *optional*, defaults to 12288):
+            Dimension of the MLP representations.
+        q_lora_rank (`int`, *optional*, defaults to 1536):
+            The rank of the query LoRA projection in MLA (Multi-head Latent Attention).
+        kv_lora_rank (`int`, *optional*, defaults to 512):
+            The rank of the key-value LoRA projection in MLA.
+        qk_nope_head_dim (`int`, *optional*, defaults to 128):
+            The dimension of the non-position encoding part of query/key heads.
+        qk_rope_head_dim (`int`, *optional*, defaults to 64):
+            The dimension of the RoPE part of query/key heads.
+        head_dim (`int`, *optional*, defaults to 64):
+            Standard dimension of qk heads, unused except for CI.
+        v_head_dim (`int`, *optional*, defaults to 128):
+            The dimension of value heads.
+        qk_head_dim (`int`, *optional*):
+            The total dimension of query/key heads. If not specified, set to `qk_nope_head_dim + qk_rope_head_dim`.
+        moe_topk (`int`, *optional*, defaults to 12):
+            Number of experts to route to for each token in the MoE layer.
+        n_routed_experts (`int`, *optional*, defaults to 512):
+            Number of routed experts in the MoE layer.
+        zero_expert_num (`int`, *optional*, defaults to 256):
+            Number of zero experts (identity function) to add to the expert pool.
+        expert_ffn_hidden_size (`int`, *optional*, defaults to 2048):
+            Hidden size of individual expert FFN layers.
+        routed_scaling_factor (`float`, *optional*, defaults to 6.0):
+            Scaling factor applied to the routing weights.
+        emb_neighbor_num (`int`, *optional*):
+            Maximum N-gram length for N-gram embeddings. This parameter determines the context window size for N-gram computation. Higher values capture 
+            longer-range lexical patterns but increase memory usage.
+        emb_split_num (`int`, *optional*):
+            Number of hash functions (or splits) to use for N-gram embeddings. Multiple hash functions help improve the quality of N-gram representations. 
+        ngram_vocab_size_ratio (`float`, *optional*):
+            Ratio multiplier for N-gram vocabulary size relative to the base vocabulary size. The N-gram vocabulary 
+            size is calculated as `vocab_size * ngram_vocab_size_ratio`. 
+
+    Example:
+    ```python
+    >>> from transformers import LongcatFlashNgramModel, LongcatFlashNgramConfig
+
+    >>> # Initializing a LongCat Flash N-gram style configuration
+    >>> configuration = LongcatFlashNgramConfig(
+    ...     emb_neighbor_num=3,
+    ...     emb_split_num=4,
+    ...     ngram_vocab_size_ratio=1.5
+    ... )
+
+    >>> # Initializing a model from the configuration
+    >>> model = LongcatFlashNgramModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "longcat_flash_ngram"
+    keys_to_ignore_at_inference = ["past_key_values"]
+    base_model_tp_plan = {
+        "layers.*.self_attn.*.q_b_proj": "colwise",
+        "layers.*.self_attn.*.kv_b_proj": "colwise",
+        "layers.*.self_attn.*.o_proj": "rowwise",
+        "layers.*.mlps.*.gate_proj": "colwise",
+        "layers.*.mlps.*.up_proj": "colwise",
+        "layers.*.mlps.*.down_proj": "rowwise",
+        "layers.*.mlp.experts.*.gate_proj": "colwise",
+        "layers.*.mlp.experts.*.up_proj": "colwise",
+        "layers.*.mlp.experts.*.down_proj": "rowwise",
+    }
+
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=131072,
+        hidden_size=6144,
+        num_hidden_layers=56,
+        num_layers=28,
+        num_attention_heads=64,
+        num_key_value_heads=None,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=1,
+        eos_token_id=2,
+        tie_word_embeddings=False,
+        rope_theta=10000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        ffn_hidden_size=12288,
+        q_lora_rank=1536,
+        kv_lora_rank=512,
+        qk_nope_head_dim=128,
+        qk_rope_head_dim=64,
+        head_dim=64,
+        v_head_dim=128,
+        qk_head_dim=None,
+        moe_topk=12,
+        n_routed_experts=512,
+        zero_expert_num=256,
+        expert_ffn_hidden_size=2048,
+        routed_scaling_factor=6.0,
+        emb_neighbor_num=None,
+        emb_split_num=None,
+        ngram_vocab_size_ratio=None,
+        **kwargs,
+    ):
+        # N-gram embedding specific parameters
+        self.emb_neighbor_num = emb_neighbor_num
+        self.emb_split_num = emb_split_num
+        self.ngram_vocab_size_ratio = ngram_vocab_size_ratio
+
+        super().__init__(
+            vocab_size=vocab_size,
+            hidden_size=hidden_size,
+            num_hidden_layers=num_hidden_layers,
+            num_layers=num_layers,
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            hidden_act=hidden_act,
+            max_position_embeddings=max_position_embeddings,
+            initializer_range=initializer_range,
+            rms_norm_eps=rms_norm_eps,
+            use_cache=use_cache,
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            rope_theta=rope_theta,
+            rope_scaling=rope_scaling,
+            attention_bias=attention_bias,
+            attention_dropout=attention_dropout,
+            ffn_hidden_size=ffn_hidden_size,
+            q_lora_rank=q_lora_rank,
+            kv_lora_rank=kv_lora_rank,
+            qk_nope_head_dim=qk_nope_head_dim,
+            qk_rope_head_dim=qk_rope_head_dim,
+            head_dim=head_dim,
+            v_head_dim=v_head_dim,
+            qk_head_dim=qk_head_dim,
+            moe_topk=moe_topk,
+            n_routed_experts=n_routed_experts,
+            zero_expert_num=zero_expert_num,
+            expert_ffn_hidden_size=expert_ffn_hidden_size,
+            routed_scaling_factor=routed_scaling_factor,
+            **kwargs,
+        )
+
+
+__all__ = ["LongcatFlashNgramConfig"]

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 3,
+  "transformers_version": "4.55.0"
+}

model-00000-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58fa5258626248582925148ffaec79f752045d76b3073eff9035f78244174e27
+size 5289947960

model-00001-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4862c4cc80ac36701781ed02a84822a152e7a285b57cf1260f8fd23209d99cce
+size 5289947968

model-00002-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:244ba1cc3f7da33f1b3bfce6eec2a38c1b1410eb1aea22c7796bd0e2bec53dd5
+size 5289946944

model-00003-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73aebcb63519a519befe68cb1c26b05bcc310ee14dfac44ca49b1f18d82c69bf
+size 5289946936

model-00004-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed6f0578b36b47680355b736a48ab732380b77d29d29f8c0e18386397523ca59
+size 5289945920

model-00005-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:411ac9baf753252971f9b7072ac08c950ff66c2acd410207a5c48480d898f532
+size 5289945912

model-00006-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:367a91bc240583a8a701b41ded917297cebd04113cdd13c3b1703be396a1fdab
+size 5289944880

model-00007-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a66b7d59d3bb94d1874dfa7fc7b91afa2632af4d2dd19b14ee503596b38f4e2
+size 5289944888

model-00008-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e10c54ec49ab87bb7abf1b9df07b2767a37a26ddf1c62014d464144c01ff8ea9
+size 5289943856

model-00009-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:99d1e0f997b7f6e4cc7c2245649f0facbcef377bf88f883a31f2faa557cd8ac9
+size 5289943880

model-00010-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f964d48cf6d02f34fc4ba242bbe1a11d83c4df5fa660695f9fee53b53b84eab
+size 5289942840

model-00011-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15b59cf9ecdb92b277d5b7a10f25f35db7373daa5050d55a65f8be2a1f283624
+size 5289942848

model-00012-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70511491f956b62c2418781e3d9db56df42510e807735ec6d8f395ccb3340e82
+size 5289941824

model-00013-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e614af5d55350bda182d21e2e39a69c4436f416f301bc536a8280d6e5fbbc465
+size 5289941816

model-00014-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ecba8f4358c5fa7a84c4e2a38b32e1c579e0601d661822288cd4ac629b58f71
+size 5289940800

model-00015-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c549541ab83b5013190c191de4db8ff7dcabc5fa78cb68796970dabda4a14636
+size 5289940792

model-00016-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3647fdc2158e2201da25a6e6b93ebc77d52e085f28995920c94346d79f5da845
+size 5289939768

model-00017-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:978483a120263813f2f0d1f0a4f741390202d0364e5c449b79e8f0fc05819a1c
+size 5289939776

model-00018-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9372820581b918bfa6866b6628013f331d91946846c65062374750cd5ca16fe9
+size 5289938744

model-00019-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c5828059c2f5e3fb067e07232856190a018863fae38a1e1858fc2dd1fb86ca1
+size 5289938752

model-00020-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94f4f1b53f863bf2e4fb7f817bfd722c4e3f712180a47b4c173efe86f9f708ff
+size 5289544376

model-00021-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae5f05ffeeeceaa0d09a4703340fd7e7834d87d23d8a7b5f928a3a767a4d29cf
+size 5289937728

model-00022-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebae9b3f3ff94b0b36b352ec72a0cc228fbe611dc59b3026a44d932961654791
+size 5289543352

model-00023-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8fee6ecd8a561d353cd135a681252f923dc247d27c624d083b45c21354d4080
+size 5289543352

model-00024-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad229f37d38cd12675367219d16f803f2a07d5fb71bc84c9d9c62598692a4100
+size 5289630936

model-00025-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:139cc01fc2445267a046da2939d589367d17cc146be8259f69bfabc9e23eca63
+size 5289630912

model-00026-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:239b6c3f708e901b273711c71f7b85619d462dcca9130e3506d6151f75b5387c
+size 5289630648

model-00027-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:982852a74d080f1567e6a02aee0b60ef56bf37f5073cf2ae904bd36561f5f395
+size 5289639576

model-00028-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c91728af1195b4b7172a2190606fd3cfb53604aef9f0a27f951af509824e740
+size 5290032856

model-00029-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cb681b76b8cf18fb5a8ae136923bcb1ff79b2d706b8e73c8677decad7ff5648
+size 5290032792

model-00030-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7298c81efffeee772390b48e182e7d033be69c1d5df86c24a54d4f68f93b726a
+size 5290032792

model-00031-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2151fa0ae1c16356c7606f7fdd0778aa49f2727f5341b7daa19ac3b8afbf6e0e
+size 5290032848

model-00032-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:590f6a7b7577a82bd7edf6434462316190aea55ccf2529053bd3465d0c232e52
+size 5289508552

model-00033-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f95d2a591d0ca4cb93db00a13c691f006ac8aaf0a4aeb09fc84447f0176aab10
+size 5289508512

model-00034-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ccc5afa878158ddab54d0e8381e4e91ecc524ebd2f99aa28cdb8e71fc6ed155
+size 5289384400

model-00035-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:022ef6dfbf2f526e0ef9a66d636ab246675815f9877f6654f45d6cda522ec9e9
+size 5289383248

model-00036-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d14972288a2e73168f1fde0536b53bd9cd4e83daa3902cd8679fce38b8f677e
+size 5290032792

model-00037-of-00038.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c9d09758a0617c2b3ecc401f55141537114cf484b549a161628751ee9b699fa
+size 5290032920

model_hashes.txt

@@ -0,0 +1,38 @@
+022ef6dfbf2f526e0ef9a66d636ab246675815f9877f6654f45d6cda522ec9e9  ./model-00035-of-00038.safetensors
+0c9d09758a0617c2b3ecc401f55141537114cf484b549a161628751ee9b699fa  ./model-00037-of-00038.safetensors
+0cb681b76b8cf18fb5a8ae136923bcb1ff79b2d706b8e73c8677decad7ff5648  ./model-00029-of-00038.safetensors
+139cc01fc2445267a046da2939d589367d17cc146be8259f69bfabc9e23eca63  ./model-00025-of-00038.safetensors
+15b59cf9ecdb92b277d5b7a10f25f35db7373daa5050d55a65f8be2a1f283624  ./model-00011-of-00038.safetensors
+1a66b7d59d3bb94d1874dfa7fc7b91afa2632af4d2dd19b14ee503596b38f4e2  ./model-00007-of-00038.safetensors
+1ccc5afa878158ddab54d0e8381e4e91ecc524ebd2f99aa28cdb8e71fc6ed155  ./model-00034-of-00038.safetensors
+2151fa0ae1c16356c7606f7fdd0778aa49f2727f5341b7daa19ac3b8afbf6e0e  ./model-00031-of-00038.safetensors
+239b6c3f708e901b273711c71f7b85619d462dcca9130e3506d6151f75b5387c  ./model-00026-of-00038.safetensors
+244ba1cc3f7da33f1b3bfce6eec2a38c1b1410eb1aea22c7796bd0e2bec53dd5  ./model-00002-of-00038.safetensors
+3647fdc2158e2201da25a6e6b93ebc77d52e085f28995920c94346d79f5da845  ./model-00016-of-00038.safetensors
+367a91bc240583a8a701b41ded917297cebd04113cdd13c3b1703be396a1fdab  ./model-00006-of-00038.safetensors
+3c91728af1195b4b7172a2190606fd3cfb53604aef9f0a27f951af509824e740  ./model-00028-of-00038.safetensors
+3ecba8f4358c5fa7a84c4e2a38b32e1c579e0601d661822288cd4ac629b58f71  ./model-00014-of-00038.safetensors
+411ac9baf753252971f9b7072ac08c950ff66c2acd410207a5c48480d898f532  ./model-00005-of-00038.safetensors
+4862c4cc80ac36701781ed02a84822a152e7a285b57cf1260f8fd23209d99cce  ./model-00001-of-00038.safetensors
+4c5828059c2f5e3fb067e07232856190a018863fae38a1e1858fc2dd1fb86ca1  ./model-00019-of-00038.safetensors
+58fa5258626248582925148ffaec79f752045d76b3073eff9035f78244174e27  ./model-00000-of-00038.safetensors
+590f6a7b7577a82bd7edf6434462316190aea55ccf2529053bd3465d0c232e52  ./model-00032-of-00038.safetensors
+5f964d48cf6d02f34fc4ba242bbe1a11d83c4df5fa660695f9fee53b53b84eab  ./model-00010-of-00038.safetensors
+70511491f956b62c2418781e3d9db56df42510e807735ec6d8f395ccb3340e82  ./model-00012-of-00038.safetensors
+7298c81efffeee772390b48e182e7d033be69c1d5df86c24a54d4f68f93b726a  ./model-00030-of-00038.safetensors
+73aebcb63519a519befe68cb1c26b05bcc310ee14dfac44ca49b1f18d82c69bf  ./model-00003-of-00038.safetensors
+7d14972288a2e73168f1fde0536b53bd9cd4e83daa3902cd8679fce38b8f677e  ./model-00036-of-00038.safetensors
+9372820581b918bfa6866b6628013f331d91946846c65062374750cd5ca16fe9  ./model-00018-of-00038.safetensors
+94f4f1b53f863bf2e4fb7f817bfd722c4e3f712180a47b4c173efe86f9f708ff  ./model-00020-of-00038.safetensors
+978483a120263813f2f0d1f0a4f741390202d0364e5c449b79e8f0fc05819a1c  ./model-00017-of-00038.safetensors
+982852a74d080f1567e6a02aee0b60ef56bf37f5073cf2ae904bd36561f5f395  ./model-00027-of-00038.safetensors
+99d1e0f997b7f6e4cc7c2245649f0facbcef377bf88f883a31f2faa557cd8ac9  ./model-00009-of-00038.safetensors
+a8fee6ecd8a561d353cd135a681252f923dc247d27c624d083b45c21354d4080  ./model-00023-of-00038.safetensors
+ad229f37d38cd12675367219d16f803f2a07d5fb71bc84c9d9c62598692a4100  ./model-00024-of-00038.safetensors
+ae5f05ffeeeceaa0d09a4703340fd7e7834d87d23d8a7b5f928a3a767a4d29cf  ./model-00021-of-00038.safetensors
+c549541ab83b5013190c191de4db8ff7dcabc5fa78cb68796970dabda4a14636  ./model-00015-of-00038.safetensors
+e10c54ec49ab87bb7abf1b9df07b2767a37a26ddf1c62014d464144c01ff8ea9  ./model-00008-of-00038.safetensors
+e614af5d55350bda182d21e2e39a69c4436f416f301bc536a8280d6e5fbbc465  ./model-00013-of-00038.safetensors
+ebae9b3f3ff94b0b36b352ec72a0cc228fbe611dc59b3026a44d932961654791  ./model-00022-of-00038.safetensors
+ed6f0578b36b47680355b736a48ab732380b77d29d29f8c0e18386397523ca59  ./model-00004-of-00038.safetensors
+f95d2a591d0ca4cb93db00a13c691f006ac8aaf0a4aeb09fc84447f0176aab10  ./model-00033-of-00038.safetensors

modeling_longcat_ngram.py

@@ -0,0 +1,338 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 Meituan
+# This code is licensed under the MIT License, for details, see the ./LICENSE file. 
+
+from typing import Optional, Tuple, Dict, List
+
+import torch
+from torch import nn
+
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.masking_utils import create_causal_mask
+from transformers.modeling_outputs import BaseModelOutputWithPast
+from transformers.processing_utils import Unpack
+from transformers.utils import auto_docstring, logging
+from transformers.models.longcat_flash.modeling_longcat_flash import (
+    LongcatFlashForCausalLM,
+    LongcatFlashModel,
+    LongcatFlashRMSNorm,
+    LongcatFlashRotaryEmbedding,
+    LongcatFlashDecoderLayer,
+    LongcatFlashPreTrainedModel,
+)
+from .configuration_longcat_ngram import LongcatFlashNgramConfig
+
+logger = logging.get_logger(__name__)
+
+
+@auto_docstring
+class LongcatFlashNgramPreTrainedModel(LongcatFlashPreTrainedModel):
+    pass
+
+
+class NgramCache(DynamicCache):
+    """
+    Extended DynamicCache for storing N-gram context alongside KV cache.
+    """
+    def __init__(self, config=None):
+        super().__init__()
+        self.ngram_context = None
+        # Keep only n-1 tokens (minimum needed for N-gram computation)
+        self.max_context_len = config.emb_neighbor_num - 1
+    
+    def update_ngram_context(self, new_tokens: torch.Tensor) -> None:
+        """
+        Update N-gram context with window management.
+        
+        Args:
+            new_tokens: New tokens to append, shape (batch_size, seq_len)
+        """
+        if self.ngram_context is None:
+            self.ngram_context = new_tokens.clone()
+        else:
+            self.ngram_context = torch.cat([self.ngram_context, new_tokens], dim=-1)
+        
+        # Truncate to maintain constant memory footprint
+        if self.ngram_context.size(-1) > self.max_context_len:
+            self.ngram_context = self.ngram_context[..., -self.max_context_len:]
+    
+    def reorder_cache(self, beam_idx: torch.LongTensor) -> "Cache":
+        """Reorder cache for beam search."""
+        # Reorder parent's KV cache
+        super().reorder_cache(beam_idx)
+        
+        # Reorder N-gram context
+        if self.ngram_context is not None:
+            self.ngram_context = self.ngram_context.index_select(0, beam_idx.to(self.ngram_context.device))
+        
+        return self
+
+
+class NgramEmbedding(nn.Module):
+    """
+    Computes embeddings enriched with N-gram features without maintaining internal state.
+    """
+    def __init__(self, config, base_embeddings):
+        super().__init__()
+        self.config = config
+        self.word_embeddings = base_embeddings
+        
+        self.m = config.ngram_vocab_size_ratio * config.vocab_size
+        self.k = config.emb_split_num
+        self.n = config.emb_neighbor_num
+        
+        self._init_ngram_embeddings()
+        self._vocab_mods_cache = None
+        
+    def _init_ngram_embeddings(self) -> None:
+        """Initialize N-gram embedding and projection layers."""
+        num_embedders = self.k * (self.n - 1)
+        emb_dim = self.config.hidden_size // num_embedders
+        
+        embedders = []
+        post_projs = []
+        
+        for i in range(num_embedders):
+            vocab_size = int(self.m + i * 2 + 1)
+            emb = nn.Embedding(vocab_size, emb_dim, padding_idx=self.config.pad_token_id)
+            proj = nn.Linear(emb_dim, self.config.hidden_size, bias=False)
+            embedders.append(emb)
+            post_projs.append(proj)
+        
+        self.embedders = nn.ModuleList(embedders)
+        self.post_projs = nn.ModuleList(post_projs)
+    
+    def _shift_right_ignore_eos(self, tensor: torch.Tensor, n: int, eos_token_id: int = 2) -> torch.Tensor:
+        """Shift tensor right by n positions, resetting at EOS tokens."""
+        batch_size, seq_len = tensor.shape
+        result = torch.zeros_like(tensor)
+        eos_mask = (tensor == eos_token_id)
+        
+        for i in range(batch_size):
+            eos_positions = eos_mask[i].nonzero(as_tuple=True)[0]
+            prev_idx = 0
+            
+            for eos_idx in eos_positions:
+                end_idx = eos_idx.item() + 1
+                if end_idx - prev_idx > n:
+                    result[i, prev_idx+n:end_idx] = tensor[i, prev_idx:end_idx-n]
+                prev_idx = end_idx
+            
+            if prev_idx < seq_len and seq_len - prev_idx > n:
+                result[i, prev_idx+n:seq_len] = tensor[i, prev_idx:seq_len-n]
+        
+        return result
+    
+    def _precompute_vocab_mods(self) -> Dict[Tuple[int, int], List[int]]:
+        """Precompute modular arithmetic values for vocabulary."""
+        if self._vocab_mods_cache is not None:
+            return self._vocab_mods_cache
+        
+        vocab_mods = {}
+        vocab_size = self.config.vocab_size
+        
+        for i in range(2, self.n + 1):
+            for j in range(self.k):
+                index = (i - 2) * self.k + j
+                emb_vocab_dim = int(self.m + index * 2 + 1)
+                
+                mods = []
+                power_mod = 1
+                for _ in range(i - 1):
+                    power_mod = (power_mod * vocab_size) % emb_vocab_dim
+                    mods.append(power_mod)
+                
+                vocab_mods[(i, j)] = mods
+        
+        self._vocab_mods_cache = vocab_mods
+        return vocab_mods
+    
+    def _get_ngram_ids(
+        self, 
+        input_ids: torch.Tensor, 
+        shifted_ids: Dict[int, torch.Tensor], 
+        vocab_mods: List[int], 
+        ngram: int
+    ) -> torch.Tensor:
+        """Compute N-gram hash IDs using polynomial rolling hash."""
+        ngram_ids = input_ids.clone()
+        for k in range(2, ngram + 1):
+            ngram_ids = ngram_ids + shifted_ids[k] * vocab_mods[k - 2]
+        return ngram_ids
+    
+    def forward(
+        self, 
+        input_ids: torch.Tensor, 
+        ngram_context: Optional[torch.Tensor] = None
+    ) -> torch.Tensor:
+        """
+        Stateless forward pass.
+        
+        Args:
+            input_ids: Current input token IDs of shape (batch_size, seq_len)
+            ngram_context: Optional historical context of shape (batch_size, context_len)
+            
+        Returns:
+            Embedding tensor of shape (batch_size, seq_len, hidden_size)
+        """
+        seq_len = input_ids.size(-1)
+        
+        # Determine complete context
+        if ngram_context is not None:
+            context = torch.cat([ngram_context[..., -(self.n-1):], input_ids], dim=-1)
+        else:
+            context = input_ids
+        
+        # Base word embeddings
+        device = self.word_embeddings.weight.device
+        x = self.word_embeddings(input_ids.to(device)).clone()
+        
+        # Precompute modular values
+        vocab_mods = self._precompute_vocab_mods()
+        
+        # Compute shifted IDs
+        shifted_ids = {}
+        for i in range(2, self.n + 1):
+            shifted_ids[i] = self._shift_right_ignore_eos(
+                context, i - 1, eos_token_id=self.config.eos_token_id
+            )
+        
+        # Add N-gram embeddings
+        for i in range(2, self.n + 1):
+            for j in range(self.k):
+                index = (i - 2) * self.k + j
+                emb_vocab_dim = int(self.m + index * 2 + 1)
+                
+                ngram_ids = self._get_ngram_ids(context, shifted_ids, vocab_mods[(i, j)], ngram=i)
+                new_ids = (ngram_ids % emb_vocab_dim)[..., -seq_len:]
+                
+                embedder_device = self.embedders[index].weight.device
+                x_ngram = self.embedders[index](new_ids.to(embedder_device))
+                
+                proj_device = self.post_projs[index].weight.device
+                x_proj = self.post_projs[index](x_ngram.to(proj_device))
+                x = x + x_proj.to(x.device)
+        
+        # Normalize
+        x = x / (1 + self.k * (self.n - 1))
+        
+        return x
+
+
+class LongcatFlashNgramModel(LongcatFlashModel):
+    """LongcatFlash model with N-gram enhanced embeddings."""
+    _keys_to_ignore_on_load_unexpected = [r"model\.mtp.*"]
+    config_class = LongcatFlashNgramConfig
+    
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.ngram_embeddings = NgramEmbedding(config, self.embed_tokens)
+
+        self.layers = nn.ModuleList(
+            [LongcatFlashDecoderLayer(config, layer_idx) for layer_idx in range(config.num_layers)]
+        )
+        
+        self.head_dim = config.head_dim  
+        self.config.num_hidden_layers = 2 * config.num_layers
+        self.norm = LongcatFlashRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = LongcatFlashRotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        **kwargs
+    ) -> BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        # Extract N-gram context if available
+        ngram_context = None
+        if isinstance(past_key_values, NgramCache) and past_key_values.ngram_context is not None:
+            ngram_context = past_key_values.ngram_context
+
+        if inputs_embeds is None:
+            inputs_embeds = self.ngram_embeddings(input_ids, ngram_context=ngram_context)
+
+        # Initialize NgramCache if needed
+        if use_cache and past_key_values is None:
+            past_key_values = NgramCache(config=self.config)
+        
+        # Update N-gram context
+        if use_cache and isinstance(past_key_values, NgramCache):
+            past_key_values.update_ngram_context(input_ids)
+
+        # Prepare cache position
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position = torch.arange(
+                inputs_embeds.shape[1], device=inputs_embeds.device
+            ) + past_seen_tokens
+
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        # Create causal mask
+        causal_mask = create_causal_mask(
+            config=self.config,
+            input_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+
+        # Forward through decoder layers
+        hidden_states = inputs_embeds
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        for decoder_layer in self.layers[: self.config.num_layers]:
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+
+        hidden_states = self.norm(hidden_states)
+        
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+            hidden_states=None,
+            attentions=None,
+        )
+
+
+class LongcatFlashNgramForCausalLM(LongcatFlashForCausalLM):
+    """LongcatFlash model for causal language modeling with N-gram embeddings."""
+    _keys_to_ignore_on_load_unexpected = [r"model\.mtp.*"]
+    config_class = LongcatFlashNgramConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = LongcatFlashNgramModel(config)
+
+    @torch.no_grad()
+    def generate(self, inputs=None, generation_config=None, **kwargs):
+        """Override to ensure NgramCache is used."""
+
+        if "past_key_values" not in kwargs or kwargs["past_key_values"] is None:
+            kwargs["past_key_values"] = NgramCache(config=self.config)
+        
+        return super().generate(inputs=inputs, generation_config=generation_config, **kwargs)
+
+__all__ = ["LongcatFlashNgramPreTrainedModel", "LongcatFlashNgramModel", "LongcatFlashNgramForCausalLM"]

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<longcat_s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</longcat_s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<longcat_pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<longcat_unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": true,
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<longcat_s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</longcat_s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 131072,
+  "pad_token": {
+    "__type": "AddedToken",
+    "content": "<longcat_pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sp_model_kwargs": {},
+  "tokenizer_class": "BloomTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<longcat_unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "chat_template": "{%- set tool_choice = tool_choice | default('auto') %}\n{%- set ns = namespace(tool_types = [], last_query_index = -1) %}\n\n{%- if tools and tool_choice != 'none' %}\n    {{- \"<longcat_tool_declare>\\n\"-}}\n    {{- \"# Tools\\n\" }}\n    {{- \"You have access to the following tools:\\n\\n\" }}\n    {%- for tool in tools %}\n        {%- if tool.type not in ns.tool_types %}\n            {%- set ns.tool_types = ns.tool_types + [tool.type] %}\n            {{- \"## Tool namespace: \" ~ tool.type ~ \"\\n\\n\" }}\n        {%- endif %}\n        {%- if tool.type == 'code_interpreter' %}\n            {%- set tool = {\"type\":\"code_interpreter\",\"function\":{\"name\":\"code_interpreter_preview\",\"description\":\"The code will be executed in a stateful Jupyter notebook sandbox environment, only supports local computation, data processing, and file operations.\\nCode sandbox environment (network isolated) Any external network requests or online API calls are prohibited.\\nIf online functionality is needed, please use other permitted tools.\\nCode will respond with the output of the execution or time out after 60.0 seconds. \",\"parameters\":{\"type\":\"object\",\"properties\":{\"language\":{\"type\":\"string\",\"description\":\"The programming language of the code to be executed. Available values: python (Default), java, go, js, ts, c, c++.\"},\"code\":{\"type\":\"string\",\"description\":\"Python code to be executed must not include the following:\\n- Importing network libraries such as requests, httplib, etc.\\n- Any form of HTTP requests.\\n- External API calls.\\n- Network port operations. Example: ```python\\nimport pandas as pd\\npd.DataFrame({'A':[1,2]})\\n```\"},\"timeout\":{\"type\":\"number\",\"description\":\"The maximum execution time of the code, in seconds. Default is 60.0.\"}}},\"required\":[\"code\"]}} %}\n        {%- endif %}\n        {{- \"### Tool name: \" + tool.function.name + \"\\n\" }}\n        {{- \"Description: \" + tool.function.description + \"\\n\\n\" }}\n        {{- \"InputSchema: \" + tool.function.parameters | tojson(ensure_ascii=False) + \"\\n\\n\" }}\n    {%- endfor %}\n    {{- '**Note**: For each function call, output the function name and arguments within the following XML format:\\n<longcat_tool_call>{function-name}\\n<longcat_arg_key>{arg-key-1}</longcat_arg_key>\\n<longcat_arg_value>{arg-value-1}</longcat_arg_value>\\n<longcat_arg_key>{arg-key-2}</longcat_arg_key>\\n<longcat_arg_value>{arg-value-2}</longcat_arg_value>\\n...\\n</longcat_tool_call>\\n' }}\n    {{- \"</longcat_tool_declare>\"-}}\n    {%- for idx in range(messages|length - 1) %}\n        {%- set msg = messages[idx] %}\n        {%- if msg.role == 'assistant' and not msg.tool_calls %}\n            {%- set ns.last_query_index = idx %}\n        {%- endif %}\n    {%- endfor%}\n{%- endif %}\n\n{%- for msg in messages %}\n    {%- if msg.role == \"system\" %}\n        {{- \"<longcat_system>\" + msg.content }}\n    {%- elif msg.role == \"user\" %}\n        {{- \"<longcat_user>\" }}\n        {%- if msg[\"files\"] %}\n            {{- '<longcat_files>\\n' ~ msg.files | tojson(indent=2) ~ '\\n</longcat_files>' }}\n        {%- endif %}\n        {{- msg.content }}\n    {%- elif msg.role == \"assistant\" %}\n        {{- \"<longcat_assistant>\" }}\n        {%- if enable_thinking == true and msg.reasoning_content and ns.tool_types != [] and loop.index0 > ns.last_query_index %}\n            {{- \"\\n<longcat_think>\\n\" ~ msg.reasoning_content ~ \"\\n</longcat_think>\\n\" }}\n        {%- endif %}\n        {%- if msg.content%}\n            {{- msg.content }}\n        {%- endif %}\n        {%- if msg.tool_calls %}\n            {%- for tool_call in msg.tool_calls -%}\n                {{- \"<longcat_tool_call>\" ~ tool_call.function.name ~ \"\\n\" -}}\n                {% set _args = tool_call.function.arguments %}\n                {% for k, v in _args.items() %}\n                    {{- \"<longcat_arg_key>\" ~ k ~ \"</longcat_arg_key>\\n\" -}}\n                    {{- \"<longcat_arg_value>\" ~ (v if v is string else v | tojson(ensure_ascii=False)) ~ \"</longcat_arg_value>\\n\" -}}\n                {% endfor %}\n                {{- \"</longcat_tool_call>\\n\" }}\n            {%- endfor %}\n        {%- endif %}\n        {{- \"</longcat_s>\" -}}\n    {%- elif msg.role == \"tool\" %}\n        {%- if messages[loop.index0 - 1].role != \"tool\"%}\n            {{- \"<longcat_user>\" -}}\n        {%- endif %}\n        {{- \"<longcat_tool_response>\" ~ msg.content ~ \"</longcat_tool_response>\"-}}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {%- if enable_thinking == true %}\n        {{- \" /think_on\" }}\n        {%- if thinking_budget %}\n            {%- if thinking_budget < 1024 %}\n                {%- set thinking_budget = 1024 %}\n            {%- endif%}\n            {{- \"\\nthinking_budget: < \" ~ thinking_budget ~ \".\"}}\n        {%- endif %}\n        {{- \" <longcat_assistant><longcat_think>\\n\"}}\n    {%- elif enable_thinking == false %}\n        {{- \" /think_off <longcat_assistant><longcat_think>\\n\\n</longcat_think>\\n\" }}\n    {%- else %}\n        {{- \"<longcat_assistant>\" }}\n    {%- endif %}\n{%- endif %}"
+}